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Abstract 


This paper reviews current research on the effectiveness of interventions and reforms that 
seek to improve the math preparedness and success of high school students entering college. 
Based on gaps in the research knowledge, it also provides recommendations for further inquiry 
in particular areas. The studies reviewed here are selected from research conducted by the 
Community College Research Center (CCRC) and the National Center for Postsecondary 
Research (NCPR), from searches of the Education Full Text database for peer-reviewed articles, 
and from searches of Google Scholar for high-quality reports. The two key criteria for inclusion 
in the review are that (1) the study in question focuses on (a) an early assessment program in 
math; (b) a math bridge, boot camp, or brush-up; (c) a reform of developmental math; or (d) 
improvements to math instruction; and that (2) at least one of the study’s outcomes is related to 
changes in math or college perfonnance. To evaluate the evidence, I report on each study’s 
design and findings. I also calculate each intervention’s effect size and categorize the effect size 
to compare impacts across the studies under review. 

Overall, the evidence is limited, but some of the interventions and reforms appear 
promising. The evidence on early assessment is minimal. The evidence on bridges, boot camps, 
and brush-ups suggests that short-term programs may only have short-term impacts. The 
evidence on different models of developmental reform varies depending on the reform model. 
For dominant models, it is positive (for compression models), insignificant (for learning 
communities), or negative (for modularization). For less prevalent models, it is positive (for 
mainstreaming) or needs further research (for statistics pathways). In terms of innovations that 
are strictly pedagogical, the strongest positive evidence is found for using structured forms of 
student collaboration and for building conceptual understanding through the use of multiple 
representations when teaching and solving problems. The evidence on computer-mediated 
instruction in the developmental math classroom is very mixed, with some studies finding 
positive effects and others finding negative effects. 
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1. Introduction 


A major challenge facing many of today’s students as they pursue a postsecondary degree 
is their lack of academic preparedness for college-level coursework and, in particular, for 
college-level math. Nationally, almost half of the 2003-04 cohort who enrolled in college 
directly after graduating from high school took at least one remedial course in college, and 
remedial course-taking is much higher at two-year colleges than at four-year colleges (see Table 
1). Examining referral to remediation at community colleges by subject, Bailey, Jeong, and Cho 
(2010) found higher remediation rates in math: 59 percent of the community college students in 
their sample were referred to developmental math, compared to 33 percent who were referred to 
developmental reading. 1 


Table 1. Number of Remedial Courses Taken Among Students Who Graduated From High School in 
2003 and Enrolled in College in 2003-04, by College Type 


Number of Remedial Courses Taken 


College Type 

0 (%) 

1 (%) 

2 (%) 

3+ (%) 

Total (%) 

Four-year 

65.4 

17.7 

8.3 

8.6 

100 

Two-year 

32.1 

20.7 

16.0 

31.2 

100 

Less-than- 


57.5 

10.6 

9.9 

22.0 

100 

two-year 

Total 

52.6 

18.6 

11.3 

17.5 

100 


Source: Author generated NCES QuickStats table using U.S. Department of Education, National Center for Education Statistics, 
BPS:2009 Beginning Postsecondary Students data. BPS: 2009 is transcript-level data for the 2003-04 cohort, tracked to 2009. 


Entering college underprepared in math has a number of consequences. First, it poses an 
obstacle to completing a college-level math course, which can be a struggle for beginning 
postsecondary students, regardless of their initial course placement in college. Transcript data on 
the 2003-04 cohort reveal how students are faring in math across two-year and four-year 
colleges (U.S. Department of Education, 2012). Over a six-year period, 21 percent of the 
students who started at a four-year college never completed a college math course, either because 
they took no math or took only a remedial math course; 44 percent completed an introductory 
college math course as their highest math course; and 35 percent completed an advanced college 
math course (for example, pre-calculus). Among students who started at a two-year college, over 
a six-year period, 5 1 percent never completed a college math course, 40 percent completed an 
introductory college math course as their highest math course, and only 9 percent completed an 

1 The data used for this study was from more than 250,000 first-time, credential seeking students who began 
their enrollment in fall 2003 to fall 2004 at one of 57 colleges participating in the Achieving the Dream initiative. 
The sample more closely represents an urban, low-income, and minority student population than do community 
colleges in the country as a whole (Bailey, Jeong, & Cho, 2010, p. 258). 
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advanced college math course. College math completion is particularly low among community 
college students assigned to remediation: Only about 20 percent of community college students 
referred to developmental math ever complete a college-level math course (Bailey et ah, 2010). 
Furthermore, there is evidence that college math success varies by student background. At the 
California community colleges, Asian and White students have significantly higher pass rates in 
college math than African American and Latino students (EdSource, 2012). 

Academic underpreparedness in math not only poses an obstacle to college math success 
but also can have an impact on an individual’s overall well-being if it hinders college 
progression and completion. The gap in earnings between high school and college graduates has 
been rising since the 1970s, and beyond their economic returns, college degrees are connected to 
many other positive outcomes, including higher levels of civic participation, healthier lifestyles, 
greater job satisfaction, and economic, educational, and health benefits that are passed down to 
one’s children (Baum, Ma, & Payea, 2010). Leaving school without basic math skills can have 
far-reaching consequences: Quantitative literacy, which includes competency in the arithmetic 
and algebraic applications that are taught in high school or developmental math, has a strongly 
predictive relationship with a young adult’s probability of employment and can explain much of 
the wage gap between African American and White young adults (Rivera-Batiz, 1992). 

Finally, math underpreparedness also impacts the number of college students who are 
able to pursue Science, Technology, Engineering, and Math (STEM) degree programs that 
require advanced college-level math. The Executive Office of the President (2012) has called for 
increasing the number of students majoring in a STEM field in order to strengthen U.S. science 
and technology industries. Yet every year, large numbers of STEM-bound students fail to persist 
in their program (U.S. Department of Education, 2012). Between 2003 and 2009, approximately 
28 percent of four-year and 20 percent of two-year college students began college as STEM 
majors. Less than half of these students persisted in a STEM major: 20 percent of four-year and 
37 percent of two-year college STEM majors left college, and 28 percent of four-year and 33 
percent of two-year college STEM majors switched to a non-STEM major. An important factor 
related to STEM attrition is preparation for college math: STEM leavers tended to take no math 
or remedial math in their first year in college, while STEM completers tended to start in college- 
level math. 

Improving the college math readiness of high school students entering college may 
contribute to decreased remediation rates and increased rates of college persistence and degree 
completion helping to improve individuals’ lifetime earnings and overall welfare. It may also 
bolster the quality of the workforce, particularly in science and technology fields — thereby 
helping to improve the nation’s economy. 
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Four Strategies for Addressing Academic Underpreparedness in Math 

Students enter college underprepared in math for a number of reasons — some did not take 
enough math in high school; some did not take the math they need for their college degree 
program; some did not master the math they took in high school; and/or some forgot the math 
they learned in high school (Fike & Fike, 2012). Additionally, students may feel frustrated or 
may struggle in their first math course in college because college placement exams are imperfect 
indicators of academic readiness; the exams misplace some students who are prepared for 
college math coursework into math remediation and misplace others who need additional support 
with basic math concepts into college-level coursework (Scott-Clayton, Crosta, & Belfield, 

2012 ). 

The underlying reasons for high rates of math remediation and college math failure thus 
necessitate a variety of reforms and interventions for improving students’ readiness for college 
math as well as for measuring incoming college students’ math skills. This review focuses on the 
interventions and reforms that postsecondary institutions currently employ to address academic 
underpreparedness in math and to foster college math success. These interventions and refonns 
fall under one of four different strategies: the first two (early assessment; bridges, boot camps, 
and brush-ups) seek to help high school students avoid math remediation before they begin their 
first semester at college so that they may enter college prepared for the first math course in their 
degree program, the third encompasses reforms to developmental math, and the fourth comprises 
improvements to math instruction in developmental and college math classrooms. Below, I 
briefly introduce the four strategies, each of which will be discussed in more detail later in the 
paper. 

Strategy One: Intervening During High School With Early Assessment 

The first strategy is the early assessment of high school students using college placement 
tests. The purpose of early assessment is to provide students an early indication of their level of 
college readiness based on the entry-level standards at their local four-year or community college 
(Barnett & Hughes, 2010). High school students who place into remediation then have time to 
work on their reading, writing, and math skills in order to avoid remediation once they enroll at 
college. 

Strategy Two: Intervening Pre-Matriculation With Bridges, Boot Camps, and 
Brush-Ups 

A second strategy is to intervene after high school students graduate but before they 
matriculate by providing short-term interventions. These bridge programs, boot camps, and 
brush-ups are designed to improve incoming college students’ math skills and help them place 
into college-level math, usually college algebra or statistics, or, for STEM students, the first 
college math course in their program, usually pre-calculus or calculus (Kallison & Stader, 2012; 
Sherer & Grunow, 2010). The interventions typically take place on college campuses, occur 
during the summer, and are targeted at students who took the placement exam in high school or 
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at the start of the summer and who placed into remedial math (or into a math class below the 
college math required for their STEM degree program). Individuals then participate in an 
intensive course for a short amount of time before the start of the semester and then retest to 
attempt to place into college-level math (or the first college math course in their program). 

Strategy Three: Reforming Developmental Math 

A third strategy is to reform developmental math in an effort to improve the outcomes of 
students who enroll in these courses. The traditional developmental education system may have 
high rates of course failure and student attrition because of its long sequence structure and 
misalignment between developmental and college-level standards and curriculum (Jaggars & 
Hodara, 2013). Reforms to developmental math attempt to address these weaknesses by 
shortening the sequence and/or aligning the curriculum to include the skills students need to be 
successful in the first college math course in their degree program. 

Strategy Four: Improving Math Instruction 

A fourth strategy is to focus on teaching and learning in the math classroom. This 
strategy overlaps in some ways with Strategy Three, given that some reforms of the 
developmental math sequence structure and curriculum may result in changes to instruction or 
explicitly include instructional refonn. However, this strategy focuses exclusively on changes to 
pedagogy inside the math classroom. I focus on math instruction in postsecondary math 
classrooms in general (including both the developmental and college level), given that some 
students who enroll directly in college math are underprepared as well and also struggle to 
succeed. 

Purpose and Method of This Review 

The purpose of this review is twofold. First, it synthesizes the research on the 
effectiveness of interventions and reforms that seek to improve the math preparedness and 
success of high school students entering college. Second, it provides recommendations for future 
research and inquiry in this area. The studies reviewed here are selected from three main sources: 
research from the Community College Research Center (CCRC) and the National Center for 
Postsecondary Research (NCPR), searches of the Education Full Text database for peer-reviewed 
articles, and searches of Google Scholar for high-quality reports. The two key criteria for 
inclusion in the review are that (1) the study focuses on (a) an early assessment program in math; 
(b) a math bridge, boot camp, or brush-up; (c) a reform of developmental math; or (d) 
improvements to math instruction; and that (2) at least one of the study’s outcomes is related to 
changes in math or college performance (e.g., employs measures of math learning, college math 
enrollment, or course pass rates, but not self-reported measures such as student satisfaction or 
math anxiety). Appendix A includes more details on the inclusion criteria and search strategy. 

To evaluate the evidence, I report on each study’s design and findings. I classify studies 
according to three types of study methods indicated in Table 2. The strongest evidence is from 
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randomized control trial (RCT) and quasi-experimental design (QED) studies because of their 
use of a comparison group that can explain what would have happened if the treatment students 
had not received the intervention. Descriptive studies provide inconclusive evidence because it is 
not possible to attribute any observed outcomes to the intervention, as it is unknown if the 
outcomes would have been similar, better, or worse in the absence of the treatment. In addition 
to identifying the design of the reviewed studies, I calculate their effect sizes and effect 
categories, which allows for standardized measures of impact and the size of the impact across 
different types of interventions. Most of the very rigorous studies reviewed here, the RCT and 
QED studies, find trivial to small positive effects of the intervention or refonn on students’ 
academic outcomes. For details on all the studies in this review, see the table in Appendix B, 
which provides a summary of each study’s findings, the effect size and/or effect category of the 
findings, and indicates if the design used non-equivalent treatment and control groups or no 
comparison group. 


Table 2: Study Type 

Method 

Description 

Randomized controlled trial (RCT) 

Participants are assigned to treatment and comparison 
groups entirely by chance. 

Quasi-experimental design (QED) 

Participants are not assigned to treatment and 
comparison groups by chance, but reasonable attempts 
are made to adjust for non-random assignment. 

Descriptive 

Study uses no comparison group or uses a comparison 
group with no adjustment for student differences. 


This review is organized as follows. For each strategy, I first describe typical 
interventions and reforms, and provide some idea of the extent of their implementation across the 
nation; next, I summarize the extent and quality of the evidence on the effectiveness of the 
interventions and reforms; and finally, I outline directions for future research. A concluding 
section summarizes the gaps in the research and suggests questions for further exploration. 


2 A trivial effect size is equivalent to a zero to 0.20 standard deviation difference in means or zero to 10 
percentage point difference in means, and a small effect size is equivalent to a 0.20 to 0.60 standard deviation 
difference in means or 10 to 30 percentage point difference in means (Hopkins, 2009). 
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2. Strategy One: Intervening During High School 
With Early Assessment 


Overview of Early Assessment 

Early assessment of high school students using college placement exams seems to be a 
fairly widespread practice across the United States (Hodara, Jaggars, & Karp, 2012). In most 
programs, colleges are minimally involved. While the colleges’ placement tests are used to 
signal high school students’ level of readiness for college-level coursework, high schools are left 
to decide what interventions, if any, to provide to students who are not deemed college-ready. 

For example, the University of Wisconsin has made a version of their placement exams available 
online for all Wisconsin high school students to take as sophomores, juniors, or seniors. The 
purpose of the early math placement tool program is twofold: to alert students to their level of 
readiness for college-level math in different degree programs at the University of Wisconsin, and 
to inform students about what kind of math they still need to take in high school to be college- 
ready in different degree programs (in order to encourage students to take more math in high 
school) (EMPT, n.d.). 

It is less common for colleges and high schools to work together to design interventions 
for students who do not test as college-ready. Because early assessments are not typically paired 
with substantive academic interventions, Tierney and Garcia (2011), who have categorized 
college readiness efforts as “(a) academic, (b) information, and (c) motivational/social” (p. 107), 
regard early assessment as informational. However, some early assessment programs have an 
academic component as well. This section of the review discusses findings on the effectiveness 
of two well-regarded early assessment programs that have an academic component. 

Evidence on Early Assessment 

The evidence on the effectiveness of early assessment is minimal — only two studies 
report any quantitative findings. The first is a QED study of the California State University 
system’s Early Assessment Program (CSU’s EAP) (Howell, Kurlaender, & Grodsky, 2010), and 
the second is a descriptive study on a college readiness protocol in El Paso, Texas (Kerrigan & 
Slater, 2010). Both studies found that early assessment reduces the need for remediation, but 
impacts are trivial in size when standardized to allow for comparisons across studies (see 
Appendix B). 

There are three components of CSU’s EAP (Venezia & Voloch, 2012). The first 
component is early testing. CSU worked together with the California Department of Education 
and California State Board of Education to set standards of college readiness in reading, writing, 
and math. CSU and K-12 faculty then revised the California high school standardized test (CST) 
to include an optional section on the CST that assesses these standards, which are essentially 
CSU entrance standards. All juniors take the CST in English and, beginning in spring 2004, have 
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the option to participate in the EAP through early testing. Students who take the CST can choose 
to complete an optional writing sample and reading test items that assess their readiness for 
college English at CSU. Only students who have completed Algebra II are eligible for the math 
CST, and they can complete optional math test items that assess their readiness for college math 
at CSU. Students’ performance on the CSU-portion of the standardized test determines if they 
are prepared for college-level English and math at CSU. If they score above a college-ready 
cutoff, they can immediately enroll in college-level courses when they start at CSU. If they score 
below, they are directed to academic supports in their senior year, which encompass the second 
component of the EAP. CSU and high school faculty collaborated in the design of English 
courses, which cover both reading and writing skills, and math courses for students who need 
additional preparation, and CSU’s diagnostic writing and math tests help students identify 
specific areas of weakness they should focus on. A third component of the EAP is professional 
development for high school teachers focused on aligning content and standards with 
postsecondary expectations. In 2008, EAP was expanded to students entering California 
community colleges, and some community colleges use student performance on the CSU portion 
of the CST to determine course placements. 

Howell et al. (2010) evaluated the impact of the first component of CSU’s EAP, early 
testing, focusing on a single campus, CSU- Sacramento, because it represents the average 
characteristics of students in the CSU system. They employed a quasi-experimental design using 
data on high school juniors in the 2001-02 to 2004-05 academic years and estimated the impact 
of the availability of EAP beginning in the 2003-04 academic year. The 2001 and 2002 juniors 
served as the control group, and the 2003 and 2004 juniors served as the treatment group. 
Controlling for student and high school characteristics, Howell et al. found that being a junior in 
the years EAP was available decreased students’ probability of referral to remedial math at CSU 
by 3.7 percentage points. This estimate represents the intent-to-treat effect because participating 
in the EAP was optional. A second model estimated the treatment-on-treated effect, the impact of 
actual participation in the EAP, with a difference-in-differences design that interacts an indicator 
of being a junior in the years when the EAP was in effect with an indicator of participating in the 
EAP. Compared to students in previous years when EAP was not available and students in years 
when EAP was available who chose not to participate, the probability of being referred to 
remedial math at CSU for students who actually completed the optional test items on the CST 

•5 

decreased by 4.1 percentage points. 

These results are robust to alternative models that attempt to address potential 
confounding factors. One concern is that the model does not account for potential changing 
conditions for juniors from 2001 to 2004 that may be related to differences in their remediation 
rate. Howell et al. (2010) addressed this concern using propensity score matching to estimate the 
probability of referral to remediation of students who were juniors in the 2003 and 2004 years, 
when the EAP was available, who chose to participate in the EAP compared to similar juniors in 

3 The intent-to-treat effect on reducing the English remediation rate is insignificant (i.e., no different from zero). 
Actual EAP participation reduces the probability of being referred to English remediation by 6.1 percentage points. 
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those same years who chose not to participate. This model indicated that participation in the EAP 
reduced the probability of being referred to remedial math by 4. 1 percentage points. A second 
concern is that the treatment on the treated estimate is upward biased because of unobservable 
differences between students who chose to participate in the EAP versus those who did not in the 
years it was available. The authors addressed this concern by estimating the intent-to-treat effect 
of the EAP on the probability of referral to remediation of students who attended high schools 
where almost all students participated in the EAP when it was available compared to students at 
those same high schools in the years before the EAP was available. This model indicated that 
being a junior in the years the EAP was available reduced students’ probability of remediation by 
3.9 percentage points. 

The Howell et al. (2010) study provides suggestive evidence that the early testing 
component of CSU’s EAP reduces the need for remediation, at least for students who attended 
CSU-Sacramcnto and were eligible for EAP because they completed Algebra II. However, no 
other quasi-experimental or experimental studies have been conducted on this intervention; thus 
the overall effect of CSU’s EAP on California students’ college readiness is unknown. 

In addition to the California study, descriptive findings from El Paso illustrate an 
association between early assessment and a small drop in remediation rates. Representatives 
from El Paso Community College (EPCC), University of Texas at El Paso (UTEP), and area 
high schools created a “college readiness protocol” in 2005 (Kerrigan & Slater, 2010). During 
their junior or senior year, high school students in El Paso participate in the protocol, which 
consists of completing the joint admissions application to EPCC and UTEP, preparing for the 
ACCUPLACER placement exam, taking the ACCUPLACER, reviewing their scores, and, if 
necessary, refreshing their skills and retaking the exam. Summer bridge programs are available 
for students who place into remediation upon retaking the exam. 

Almost three-quarters of incoming EPCC freshmen in 2007 participated in the protocol in 
high school, and EPCC institutional research found that 5 percent placed into college-ready math 
compared to 3 percent of the 2005 class, who had not been exposed to the protocol in high 
school (Kerrigan & Slater, 2010). Additionally, compared to the 2005 cohort, a much greater 
proportion of the 2007 cohort placed into the highest developmental math course, and a lower 
proportion placed into the lower levels. These findings do not account for differences in student- 
level characteristics, time-varying confounds, and endogenous sorting. For example, perhaps the 
2007 cohort had better math skills, on average, than the 2005 cohort; perhaps high schools 
improved over time; or perhaps only the most motivated students took part in the college 
readiness protocol, inflating the true impact of early assessment. As a result, the association 
between early assessment in El Paso and the 2 percentage point increase in the proportion of 
college-ready students from 2005 to 2007 may be entirely explained by other factors. 
Furthermore, the fact that only 5 percent of incoming EPCC students placed into college math 
suggests that more far-reaching, systemic changes need to be considered to have a substantial 
impact on the college math readiness of El Paso high school students. 
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Directions for Research on Early Assessment 

The two reviewed studies, as well as informal reports from other colleges that have 
conducted internal research on their remediation rates before and after implementation of early 
assessment (Hodara et ah, 2012), suggest that early assessment may increase the number of high 
school students placing directly into college-level coursework. However, beyond this finding, we 
know little about early assessment and its impact on students. Therefore, rigorous research needs 
to be conducted to confirm if early assessment has a direct effect on reducing the remediation 
rate across different models, in different contexts, and with different types of students. A deeper 
look at the long-term impact of early assessment programs would also identify if they have more 
substantive benefits beyond reducing the remediation rate; in particular, in-depth studies might 
indicate whether programs enhance students’ perfonnance in math courses, college progression, 
persistence, and degree attainment. 

Future research in this area should also explore the mechanisms underlying the impact of 
early testing. Howell et al. (2010) suggested two potential reasons for the impact of early testing 
on lowering students’ probability of referral to remediation at CSU. One potential explanation is 
that high school juniors referred to remediation based on the CST were discouraged and, as a 
result, decided not to apply to CSU, thus lowering the remediation rate. However, Howell et al. 
found no evidence that students who did not test as college-ready on the CST were less likely to 
apply to CSU. Thus, Howell et al. inferred an alternative explanation: that juniors referred to 
remediation based on the CST took the necessary steps to improve their math skills, such as 
participating in an extra math class or academic support in their senior year, which helped them 
avoid remediation when they eventually enrolled at CSU. 

Future research could also undertake an exploration of the role of early assessment in 
strengthening collaborations between secondary and postsecondary systems. One hypothesis is 
that early assessment programs have contributed to stronger relationships between secondary and 
postsecondary systems. Collaborative efforts between high school and college stakeholders 
working to implement early assessment programs have been documented in a number of states 
(California in Knudson, Zitzer-Comfort, Quirk, & Alexander, 2008; Spence, 2005; Spence, 

2009; Venezia & Voloch, 2012; Florida in Bilsky, 2011; Burdman, 2011; Collins, 2009; New 
York in Venezia & Voloch, 2012; and Texas in Kerrigan & Slater, 2010). 

For example, in 2007, after Florida joined the American Diploma Project, an initiative of 
Achieve, Inc. that works to help states raise academic standards, over 100 teachers and faculty 
from high schools, two-year colleges, and state universities from across the state met to discuss, 
decide on, and then draft entry-level college competencies in reading, writing, and math, called 
the Postsecondary Readiness Competencies (PRCs) (Bilsky, 2011). Then, in 2008, the Florida 
State Legislature passed legislation requiring all high school juniors to be assessed using a 
statewide college placement exam, and requiring high schools to provide students who place into 
remediation with remedial coursework prior to graduating from high school (Collins, 2009). The 
legislation prompted the faculty group who drafted the PRCs to reconvene and become heavily 
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involved in the development of the new placement exam (Bilsky, 2011). The faculty group 
reviewed test items submitted by test vendors to assess which were the most closely aligned with 
the PRCs in order to help choose a test vendor to develop the new placement test, eventually 
called the Postsecondary Education Readiness Test (PERT). The faculty group also developed 
sample test items based on the PRCs for the PERT, and when the Common Core State Standards 
were released, the faculty group worked to align the PRCs with these standards and develop and 
review new test items for the PERT. In 2010, Florida began assessing high school juniors using 
the PERT and offering transitional courses to seniors who placed into remediation based on their 
PERT performance. 4 

However, an alternative hypothesis is that comprehensive early assessment programs, 
like the one in Florida, that involve an informational and academic component and are related to 
or result in efforts to align high school graduation and college entry standards are rare. Instead, 
qualitative studies suggest that while early assessment efforts are common, there is a lack of 
deeper collaboration between secondary and postsecondary institutions across the country. In a 
study of assessment and placement practices across 38 two-year colleges in seven states, Hodara 
et al. (2012) found a large number of early assessment efforts, but at only one college were 
faculty working with local high school administrators and teachers to create remedial 
interventions at the high school level for juniors who tested into remediation based on the college 
placement exam. Additionally, across the seven states, only Texas was involved in aligning high 
school graduation requirements with college -readiness standards (Hodara et al., 2012). In 
Grubb’s (2013) study of 13 community colleges in California, only one college was found to be 
collaborating with a feeder high school to discuss ways to improve student readiness for college 
and to lower rates of remedial placement. 

Partnerships between colleges and local high schools may become more common with 
the implementation of the Common Core State Standards (CCSS) (Barnett & Fay, 2013). For 
example, the implementation of the standards has spurred conversations in some postsecondary 
systems about aligning developmental education and/or introductory math and English curricula 
to the CCSS. In addition, early assessment may become more widespread. In the 2014-15 
academic year, new assessments aligned with the Common Core States Standards will be 
administered to juniors to assess their level of college and career readiness in the 45 states that 
have adopted the standards. A cutoff score will be set that represents the minimum score to be 
considered college- and career-ready. If colleges honor this cutoff and allow students who score 
at or above it to enroll in college coursework without taking the college placement exams, then 
the administration of the new assessments will essentially represent a large-scale early 
assessment program. The scale of implementation poses a motivation for studying if and how 
early assessment affects students and presents opportunities for future research in this area. 


4 To date, there is no descriptive or quasi-experimental research on the impact of the administration of the 
Florida PERT and remedial interventions at the high school level on student outcomes. 
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To summarize, I recommend two general directions for research on early assessment: 

1. Rigorous evidence on the effects of early assessment and the 
mechanisms underlying its effects. Research questions may 
include: 

a. What is the causal effect of signaling to high school 
students that they are not college-ready on their enrollment 
and performance in college math, and what are the long- 
term impacts on college outcomes, such as progression, 
persistence, and degree attainment? 

b. How do the effects vary across different contexts and 
models of early assessment, for example, when early 
assessment takes place at the sophomore, junior, or senior 
level; for students with different levels of initial math 
preparation in high school; and when early assessment 
programs are purely informational versus when they have 
an academic component and/or a motivational/social 
component? 

c. What is the effect of early assessment on participation in 
remedial interventions and other coursework choices at the 
high school level? 

d. For students who do not place as college-ready based on 
early assessment measures, how effective are remedial 
interventions at the high school level in improving their 
college math readiness, measured by math placement exam 
performance and enrollment and perfonnance in college 
math? What features of remedial interventions at the high 
school level make them more or less effective? 

2. An understanding of the role early assessment plays in 
strengthening secondary and postsecondary partnerships. 

Research questions may include: 

a. Do early assessment initiatives lead to greater collaboration 
between secondary and postsecondary institutions? What is 
the nature of this collaboration? Does early assessment lead 
to more remedial interventions at the high school level? 

Revisions of placement exams? More testing at high 
schools? 
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b. How will the implementation of the Common Core State 
Standards assessments impact early assessment programs 
and secondary and postsecondary partnerships generally? 


3. Strategy Two: Intervening Pre-Matriculation With 
Bridges, Boot Camps, and Brush-Ups 

Overview of Bridges, Boot Camps, and Brush-Ups 

Bridges, boot camps, and brush-ups typically recruit high school students in their senior 
year who are on track for remediation or individuals who apply to college early and place into 
remediation. The programs consist of intensive basic skills courses that take place during the 
summer on college campuses (Sherer & Grunow, 2010). According to recent national scans of 
“math intensives” and developmental education innovations, bridges and boot camps are 
relatively common at colleges across the country (Edgecombe, Cormier, Bickerstaff, & 
Barragan, 2013; Sherer & Grunow, 2010). Community colleges also seem to be increasingly 
offering brush-up courses because they are low-cost and easy to scale up (Hodara et al., 2012). 

Summer bridge programs have a long history in higher education (Strayhorn, 2011). 
Some bridge programs are for students who place into remediation while others are for students 
entering STEM programs who are not prepared for the first college math course in the STEM 
program, often pre-calculus or calculus. Many bridge programs are targeted to minorities and 
first-generation college students, and to individuals who have lower rates of persistence and 
completion than other college students and who are underrepresented in the STEM fields 
(Kallison & Stader, 2012; Lenaburg, Aguirre, Goodchild, & Kuhn, 2012; Stolle-McAllister, 
2011; Walpole et al., 2008). Bridge programs are typically designed to improve students’ math 
skills, as well as orient them to college culture, build their study skills, and provide them with an 
important network of support prior to enrolling. In this way, bridge programs attempt to impact 
students’ college outcomes through multiple mechanisms: academic supports to help students 
start college in college-level coursework and non-academic supports intended to improve their 
persistence in college (Barnett et al., 2012). 

Bridge programs and boot camps are very similar, and sometimes colleges use the term 
bridge program and boot camp interchangeably, but boot camps tend to be shorter in duration 
than bridge programs and primarily focus on building students’ math skills and improving their 
performance on placement tests (Sherer & Grunow, 2010). 
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In contrast to bridge programs and boot camps, brush-up courses tend to last only a few 
hours and focus exclusively on placement test preparation, either for students testing for the first 
time or for students who initially tested into remediation and want to retest (Hodara et ah, 2012). 
These courses are not necessarily designed to improve math skills; rather, they are focused on 
improving the accuracy of the student’s test score and associated placement, by ensuring that the 
student is familiar with the exam and has the opportunity to refresh relevant pre-existing math 
knowledge. While bridge programs and some boot camps tend to have an infonnational, 
academic, and motivational or social component, brush-ups only have an academic component. 

Evidence on Bridges, Boot Camps, and Brush-Ups 

The evidence on bridges, boot camps, and brush-ups includes one RCT (Barnett et ah, 
2012) and a set of descriptive studies, many of which report institutional research (Gleason et ah, 
2010; Hodara et al., 2012; Kallison & Stader, 2012; Reisel, Jablonski, Hosseini, & Munson, 
2012; Sherer & Grunow, 2010). Overall, these studies do not yield strong findings in favor of 
bridges, boot camps, and brush-ups: impacts are largely questionable given the lack of an 
equivalent comparison group and range in size and sign from trivially negative to moderately 
positive (see Appendix B). 

Barnett et al. (2012) conducted a randomized experiment of developmental summer 
bridge programs offered to recent high school graduates at two open-admissions four-year 
colleges and six community colleges across the state of Texas. Students attended the summer 
bridge programs for three to six hours daily for four to five weeks. Although the programs at 
each college varied, they shared similar features: accelerated instruction in math, reading, and/or 
writing; academic support outside of class through individual tutoring and/or access to labs 
where tutors were available; a college knowledge component provided through advisors, 
mentors, and financial aid staff through a college success course or more informal presentations 
and workshops; and a stipend of $150 delivered at the start of the program to improve 
recruitment efforts and $250 delivered after completion of the program to encourage students to 
complete. 

Study participants included those who placed into remediation, usually based on 
placement testing in their senior year in high school, and applied for admission to the summer 
bridge program in the summer of 2009 (Barnett et al., 2012). After agreeing to take part in the 
study, participants were randomly assigned to the summer bridge treatment group or the control 
group. Therefore, students in the treatment and control groups should be similar, on average, 
across observable characteristics, such as demographics and placement scores, and unobservable 
characteristics, such as motivation, and, therefore, any differences in their outcomes can be 
attributed to participation in the summer bridge program. 

Students were followed for two years, and differences in outcomes reveal that the 
summer bridge programs had extremely modest impacts (Barnett et al., 2012). A higher 
proportion of students in the treatment group than the control group passed a college-level math 
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course through the fall 2010, but by the spring 2011 there was no significant difference in the 
proportion of treatment group students (46.5 percent) and control students (43 percent) who 
passed college math. 5 Differences in persistence (i.e., total semesters registered over the two- 
year period), the number of college credits earned, and the number of college credits attempted 
of students in the treatment and control groups were also not statistically different from zero. The 
limitations of the study are that the researchers could not disentangle the effects of different 
components of the summer bridge program, and the results pertain to a specific group of students 
that placed into remediation but were motivated enough to sign up for a summer bridge program, 
and therefore the findings are not generalizable to all summer bridge programs and all types of 
students. 

Importantly, Barnett et al. (2012) did not examine the program’s impact on the most 
common outcome across studies of early assessment, bridges, boot camps, and brush-ups: course 
placement after the intervention. Instead, the researchers focused on the longer-term outcome of 
eventual college math completion. The statistically similar college math pass rates of the 
treatment and control groups signify that even if the Texas summer bridge program improved the 
college math placement of treatment students so that more treatment students placed into college 
math than control students, it had no impact on actually helping them pass college math. This 
suggests that programs that focus on improving placement test performance may not have a 
meaningful impact on improving students’ performance in college math or long-term outcomes, 
such as persistence and progression. 

Kallison and Stader (2012) also conducted a study on the Texas developmental summer 
bridge programs, but their study was descriptive and has limited internal validity. They examined 
gains in placement test scores among summer bridge participants who took a pretest and posttest. 
Only four institutions collected math test score data, and at these colleges there was no statistical 
difference in the math placement exam performance of students before and after participating in 
the summer bridge program. 

Two descriptive studies (Gleason et al., 2010; Reisel et al., 2012) report on the outcomes 
of students in bridge programs at University of Alabama and University of Wisconsin- 
Milwaukee targeted to incoming STEM majors who placed below calculus; both universities 
identified an initial placement below calculus as a key predictor of STEM attrition. The bridge 
programs provided accelerated instruction in pre-calculus mathematical content, including 
college algebra and trigonometry, so that students did not delay enrollment in the calculus 
sequence. Both programs had similar features: they were residential (although in the first two 
years the University of Wisconsin program also allowed students to complete the program online 
from home); they lasted about one month; and students spent the mornings learning math content 
and the afternoons applying their math skills in real-world problem-solving activities. The 
University of Alabama program was voluntary for students who scored in a certain range below 


5 The same is true for treatment and control students referred to developmental reading: there were no significant 
differences in the proportion of students enrolling in college-level English. 
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the cutoff for placement into calculus, while the University of Wisconsin program was 
mandatory for students whose math ACT, high school grades, and class rank matched the profile 
of engineering students with the lowest graduation rates. 

Gleason et al. (2010) found that although the Alabama students who took part in the 
summer bridge program, from 2005 to 2007, significantly increased their math placement scores 
upon completion of the program, not all students were able to place into calculus in the fall. 
Program students who placed into pre-calculus subjects performed significantly worse in pre- 
calculus courses than students with similar placement scores who placed directly into these 
courses, and there was no significant difference in the grades of program students who placed 
into calculus and control students who placed directly into calculus. The meaning of these results 
is unclear, because it is unknown how the program students would have fared in their first- 
semester math course in the absence of participating in the summer bridge program. 6 In terms of 
retention, 92 percent of students in the pilot program were still in a STEM degree program three 
years later, compared to 63 percent of STEM students who entered in the same year with similar 
placement test scores. Although this difference in STEM retention is quite large, it could be due 
entirely to the higher motivation of students who chose to participate in the summer bridge 
program. 

The Reisel et al. (2012) study of the University of Wisconsin program did not employ a 
comparison group, and its purpose was primarily to track the relationship between features of the 
program and gains in students’ math placement test scores upon completion of the program. 
Students in the residential program improved their math placement scores more than students in 
the online program (prompting the university to discontinue the online option in 2009); gains 
were trivial for students in the online program but of a moderate size for students in the 
residential program. In addition, in the morning session, students worked through individualized, 
self-paced math content using ALEKS (Assessment and Learning in Knowledge Spaces), a 
computer-adaptive, online assessment and learning program, and students with more ALEKS 
time-on-task had greater improvements in their placement test scores. 

The remaining findings on boot camps and brush-ups are all descriptive reports based on 
colleges’ internal research. Boot camps at EPCC in Texas, LaGuardia Community College in 
New York, Montgomery County Community College in Maryland, and Pasadena City College in 
California all report that a proportion of participants improved their placement exam 
performance and placed into a higher developmental math level or college-level math (Sherer & 
Grunow, 2010). At EPCC, specifically, students participating in the PREP boot camp take a 
diagnostic test on A+ Advancer, an online resource aligned to the ACCPUPLACER placement 
exam that creates an individualized test prep course for each student that targets his or her own 
areas of weakness. Students complete the individualized course in a lab where there are tutors 

6 For example, in the absence of the program, would program students fare even more poorly in calculus? If so, 
then the observed null result for calculus performance is in fact a positive result. On the other hand, if these students 
would have performed just as well in calculus without the program, then the null result indicates that the program 
was not helpful. 
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available. After they retake ACCUPLACER, they meet with a counselor to select courses and 
discuss their goals. From 2003 and 2008, between 52 to 66 percent of PREP participants placed 
at least one level higher in math (Sherer & Grunow, 2010). However, without a comparison 
group, it is unknown what the outcomes of these students would be in the absence of the 
treatment — for example, if the students had simply retested without participating in the boot 
camp. 

Institutional research from a North Carolina college also finds a positive association 
between test preparation in a brush-up course and improvements in placement (Hodara et ah, 
2012). The North Carolina college created an online course for their reading, writing, and math 
placement exams, which students can access and complete from any computer at their 
convenience. The online course opens with a video of a student explaining the importance of the 
placement exams and the content of the review. For each subject, the course includes 
approximately one hour and twenty minutes of content: a diagnostic pretest, infonnation on areas 
where the student is weak, instructional videos that cover the test content, a posttest, and 
additional resources to help students prepare for the test, such as PowerPoint presentations 
created by faculty and links to ACT online practice materials. The college found that from fall 
2010 to spring 2011, among all students who took the review course before re-testing, 35 percent 
of students tested at least one level higher in the developmental math sequence and had similar or 
higher pass rates in the courses they retested into, compared with their counterparts who placed 
directly into the course. Currently, the state is working with a new placement test vendor to 
create similar, online brush-up courses that will be offered to incoming community college 
students statewide. 

Directions for Research on Bridges, Boot Camps, and Brush-Ups 

There is only one extant rigorous study on the impact of a summer program that attempts 
to improve the math placement and college performance of high school students who are on 
track for remediation. Barnett et al. (2012) found that the Texas developmental summer bridges 
programs had only short-tenn impacts on passing introductory college math, suggesting that 
short-term interventions are not sufficient to impact the educational outcomes of students in a 
meaningful way. In other words, summer programs should be one small component of a larger 
support framework for high school students as they transition from high school to college; other 
components might include early assessment, remedial interventions at the high school level, and 
supports that continue throughout a students’ college career. Supporting students from their high 
school years into college could be an extremely resource-intensive, yet profoundly important 
endeavor. Therefore, understanding which program features have a long-term impact on students 
and are cost-effective is an important direction for research. 

Some of this research is presently under way. Specifically, MDRC is conducting an 
evaluation of the “Getting Ready for Success” (GRS), a program in Tacoma, Washington, for 
low-income high school juniors, which builds on the lessons of the Texas developmental summer 
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bridges evaluation (Sepanik, 2012). GRS students participate in an academic enrichment 
program before their senior year, work with college advisors and mentors during their senior 
year, and if they still need remedial support, take part in a summer bridge program after 
graduating from high school. During their first year in college, they continue to receive advising 
and mentoring, and receive incentives along the way that total to $3000 if they successfully 
complete all components of the program. 

Another direction for future research is to better understand the causal effects of 
placement exam preparation, primarily offered through boot camps and brush-ups, on not only 
traditional indicators of placement test performance but also on math learning, as measured by 
grades in college math and perhaps a learning assessment, and subsequent college outcomes. 

Test preparation seems to help students who are underplaced and do not need remediation by 
allowing them to avoid unnecessary developmental coursework (Hodara et ah, 2012). Test 
preparation may effectively prepare students for college math because it helps students refresh 
their math skills and/or relearn the math content. However, an alternative scenario is that 
improvements in placement exam perfonnance do not necessarily mean students are prepared for 
college-level math, because students may be learning how to perform better on the exam but not 
learning the underlying content. If so, then students may need additional supports to perform 
well in the higher level course they place into after successfully participating in a boot camp or 
brush up. 

Improvements in placement exam performance may also be weakly related to 
performance in college coursework because commercial placement exams comprise standardized 
content that is not necessarily aligned to the institution’s introductory college-level math and 
English curricula (Hodara et al, 2012). The challenge in developing placement exams that more 
accurately reflect incoming students’ readiness for introductory college coursework is creating 
an exam that directly tests the skills students need to be successful in their colleges’ introductory 
math and English courses. Creating a customized exam may be a particular challenge for 
resource-constrained institutions and systems; commercial exams (for example, COMPASS and 
ACCUPLACER) are readily available and are perhaps less expensive than creating a customized 
exam. Yet, increasingly, higher education systems are addressing the misalignment between the 
content of the placement exam and introductory college English and math by developing their 
own customized exams that are aligned to the system’s standards for readiness in introductory 
college English and math. 7 An interesting and useful research question would be to investigate if 
customized placement exams are more effective at predicting students’ readiness for college than 
commercial exams. 

To summarize, I recommend two general directions for research on bridges, boot camps, 
and brush-ups: 


7 These are some systems and states that have a customized exam or are developing one: University of 
Wisconsin, City University of New York in English, Virginia Community College System, Florida, North Carolina 
Community College System (in development), and Texas (in development). 
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1 . An exploration of cost-effective features and models of bridges 
and boot camps that lead to long-term benefits for students. 

Research questions may include: 

a. What academic and non-academic features of bridges and 
boot camps are most important in supporting students’ 
college persistence and degree attainment? 

b. What is the optimal length of time that programs should 
support students as they transition from high school to 
college in order to yield cost-effective, long-tenn impacts? 

2. Causal evidence on the effects of test preparation delivered 
primarily through boot camps and brush-ups. Research 
questions may include: 

a. What are the causal effects of test prep courses on 
enrollment and performance in college math courses as 
well as on performance on a math learning assessment? 

b. How do the effects differ if instruction is delivered face-to- 
face versus self-paced/online? How do the effects differ if 
features of the intervention include non-academic resources 
and staff support? How do the effects differ for customized 
versus commercial placement exams? 


4. Strategy Three: Reforming Developmental Math 

Overview of Developmental Math Reforms 
The Status Quo 

Typically, developmental math encompasses a sequence of courses that students are 
placed into based solely on their placement exam performance (Hodara et ah, 2012). Sequences 
range in length, and some colleges may require more remedial math for students entering a 
STEM field compared to a liberal arts field. In a three-course sequence, for example, the lowest 
math course might cover arithmetic, the next course introductory algebra concepts, and the 
highest level course intermediate algebra, intended to prepare students for their first college-level 
math course, often College Algebra or Statistics. 
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The majority of developmental math students never complete their remedial 
requirements; some do not do so due to course failure, and others because they simply do not 
enroll in the next course in the sequence (Bailey et ah, 2010). In general, these students have 
lower credit accrual, persistence, and graduation rates in comparison to their college-ready peers 
(Attewell et ah, 2006; Jaggars & Hodara, 2011; Jenkins, Jaggars, Roksa, Zeidenberg, & Cho, 
2009). Yet, it is difficult to infer if these poor outcomes are due to the ineffectiveness of 
developmental education or to the pre-existing characteristics of students who are in 
developmental education courses. 

A growing body of rigorous research is attempting to identify the causal impact of 
developmental education on students’ outcomes by disentangling the impact of student-level 
characteristics from the impact of developmental education (Bettinger & Long, 2009; Boatman 
& Long, 2010; Calcagno & Long, 2008; Dadgar, 2012; Hodara, 2012; Martorell & McFarlin, 
2011; Scott-Clayton & Rodriguez, 2012). This research primarily relies upon regression 
discontinuity (RD) to address the non-random sorting of students into developmental and 
college-level courses. RD is used when an arbitrary cutoff (in this case, a score on a college 
placement exam) determines if individuals are assigned to an intervention or not (in this case, 
developmental or college courses). The underlying assumption of RD is that while placement 
exam performance may be positively related to educational outcomes, this relationship should be 
smooth (Imbens & Lemieux, 2008). For example, as placement score performance increases, we 
might see a smooth trend upward in the proportion of students enrolling in and passing college 
math. Additionally, the relationship between placement exam performance and student 
characteristics should be smooth. If these assumptions are met, then any discontinuity, or 
unnatural jump, in outcomes coincident with the score cutoff can be directly attributed to the 
only change that occurred at the cutoff — assignment to developmental or college math — rather 
than to student characteristics and other confounding factors, both observable and unobservable. 

RD studies on developmental math at public four-year and community college systems 
across the county (Florida, New York, Virginia, and Texas) have largely found that 
developmental math has an insignificant or negative effect on the educational outcomes of 
students (Calcagno & Long, 2008; Dadgar, 2012; Martorell & McFarlin, 2011; Scott-Clayton & 
Rodriguez, 2012). Similar findings have emerged from studies of students assigned to lower 
levels versus higher levels of developmental math: around lower cutoffs, students tend to not 
benefit from more semesters of developmental math (Boatman & Long, 2010; Dadgar, 2012). 

The consistency of these findings provides compelling support for their internal validity, 
but ultimately they do not identify exactly what is wrong with developmental education. The 
findings may be the result of misplacement: If the cutoffs are set too high or the placement 
exams are not accurate measures of math readiness, then some students placed into remediation 
may not actually need it. Or, perhaps students’ developmental education placement is 
appropriate, but the long sequence length decelerates their progress into college coursework and 
they never catch up; perhaps the curriculum is not tied to what students need to know and be able 
to do to be successful in the college math for their own degree programs; and/or perhaps the 
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courses’ pedagogy does not effectively promote students’ math learning and understanding. 
Across the country, colleges have sought to address each of these potential shortcomings of 
developmental education with a variety of reforms in an effort to improve students’ 
developmental and college math success (Edgecombe et ah, 2013b). These reforms, as well as 
those under strategy four, are focused primarily on academics although many have an explicit 
motivational or social component as well. 

Alternatives to the Status Quo 

In a recent scan of developmental education innovations that changed the developmental 
curricula, course structure, and/or pedagogy, Edgecombe et al. (2013b) found 52 community 
colleges engaged in developmental reform, with 41 innovations focused on math, 21 focused on 

o 

English (combined reading and writing) and/or reading, and 4 that focused on math and English. 
These reforms are aimed at changing the developmental course structure and/or curriculum, 
often in an effort to accelerate student progress through developmental education, and some 
attend to pedagogical changes as well. Edgecombe et al. identified how common each reform 
type was across the 52 colleges, and in the following subsections, I describe dominant and less 
prevalent models of reform and then present evidence on what is known about the effectiveness 
of these reforms. 

Dominant models of developmental reform. Edgecombe et al. (2013b) found that the 
most common developmental innovations were compression (occurring at 17 colleges in their 
scan), modularization (occurring at 16 colleges), and linked courses or learning communities 
(occurring at 16 colleges). Compression and modularization are intended to accelerate student 
progress through developmental education by changing the traditional sequence structure and/or 
curriculum and offering only the content students need to succeed in college math (Edgecombe, 
2011). Compression shortens the developmental sequence by combining two or more sequential 
developmental courses into a single semester. A compressed course does not require a change of 
curriculum if the number of contact hours remains the same, but if the number of contact hours 
decreases, then the content must be redesigned as well, often by eliminating redundancies in 
curriculum. Modularization divides the developmental curriculum into modules or single units 
that represent discrete math learning outcomes or competencies. Under this model, certain 
modules may be required for some degree programs but not others, and students may be allowed 
to move from one module to the next at their own pace using computer-mediated instruction. The 
course structure is not altered if the modules are offered within the existing semester-long 


s The sample of colleges resulted from a detailed scan of developmental innovations occurring across the 
country, but do not represent all colleges innovating across the country. Edgecombe et al. (2013b) identified 
developmental education innovations through: phone conversations with reform leaders; online searches of 
professional associations and networks, conference proceedings, the RP Group website, and the Getting Past Go 
blog; the list of previous winners of the League for Innovation in the Community College’s annual competition 
posted on their website; and recommendations from faculty, administrators, and other researchers familiar with 
developmental education reform landscape. 
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course, but it is substantially changed if colleges offer the modules as stand-alone, short courses, 
as the Virginia community colleges do under their developmental math redesign (Asera, 2011). 

Linked courses or learning communities have a longer history at community colleges 
than models of acceleration. In a learning community, students enroll in two or more courses in 
the same semester together in order to create a supportive network of peers, shared learning 
experiences, and deeper connections to college (Tinto, 1997). The cohort experience is thought 
to improve student learning and persistence. 

Less prevalent models of developmental reform. Less common developmental refonns 
include curricular redesign and mainstreaming (Edgecombe et ah, 2013b). One curricular math 
reform that is growing in popularity revises the traditional algebra-based developmental math 
curriculum and shortens the sequence by creating a statistics-based developmental math 
sequence for non-STEM students. The Carnegie Foundation for the Advancement of Teaching 
set out to address the high rates of course failure and attrition among developmental math 
students by designing a developmental math pathway that is more closely aligned with the skills 
students need to be successful in college statistics, the college math course most liberal arts 
students take (Cullinane & Treisman, 2010). They first developed Statway, a year-long statistics 
pathway with one semester of developmental statistics and one semester of college statistics, that 
30 community colleges across the country have adopted. More recently they developed a one- 
semester developmental quantitative literacy course, called Quantway, that eight community 
colleges have adopted. 9 Both pathways focus on teaching mathematics content that can be 
applied to solve everyday problems; this type of content lends itself to project-based learning and 
tends to be more closely aligned with the skills liberal arts students need to be successful in their 
degree programs and careers (Cullinane & Treisman, 2010; Merseth, 201 1). The courses also 
seek to promote “productive persistence” (Merseth, 201 1, p. 36), or non-academic behaviors 
associated with time management, motivation, and self-efficacy, through integrating college 
knowledge and other supports into the curriculum. In addition to the Carnegie Foundation- 
affiliated colleges, seven community colleges that are part of the California Acceleration Project 
have developed their own statistics pathways that are similar to Statway. 10 

Another acceleration model involves mainstreaming students who place into 
developmental math into college-level math courses and providing them with a supplemental 
support course (Edgecombe, 2011). Mainstreaming is more common in English than in math, 
and overall, a much less common form of acceleration compared to compression and 
modularization (Edgecombe et ah, 2013b). Mainstreaming usually requires a change of 
placement test score policy to allow students who otherwise would have been required to take 
developmental education to enroll in college coursework. Additionally, faculty may develop or 
link a supplemental support course to their college-level courses designed specifically for 


9 See description on the Carnegie Foundation website: http://www.carnegiefoundation.org/statway 

10 See California Acceleration Project website: http://cap.3csn.org/developing-pilots/pre-statistics-courses/ 
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students who would have been in developmental education but who are now in college 
coursework. 

A second form of mainstreaming is integrating basic skills instruction into college-level 
courses (Perin, 2011). Examples of integration primarily come from certificate and associate 
occupational degree programs rather than liberal arts and sciences degree programs, and so 
usually involve integrating adult basic education curriculum into college-level occupational 
courses. For example, the Career Pathway Bridge programs at the Wisconsin technical colleges 
integrate different basic skills subjects into college-level courses for students seeking certificates 
in manufacturing, automotive fields, welding, baking, and nursing, as well as some associate 
degrees in similar fields (Valentine & Pagac, 2010). 

Evidence on Developmental Math Reforms 

Dominant Models of Developmental Reform 

In this section, I summarize evidence from one RCT (Weissman, Butcher, Schneider, 
Teres, Collado, & Greenberg, 2011), three QED studies (Boatman, 2012; Edgecombe, Jaggars, 
Baker, & Bailey, 2013; Hodara & Jaggars, 2013), and three descriptive studies (Speckler, 2008; 
Squires, Faulkner, & Hite, 2009; Twigg, 2005). The research suggests promising, yet short-term 
impacts of compression, mild impacts of learning communities, and unclear impacts of 
modularization; all effect sizes are in the trivial to small size range (see Appendix B). 

Compression. I summarize two QED studies here. Although the first study is not about a 
compression reform, it examines the same underlying treatment: shorter versus longer 
developmental math sequences. Hodara and Jaggars (2013) exploited a natural experiment in 
which community colleges in the same system, the City University of New York (CUNY) 
community college system, have the same standards of college readiness but different-length 
sequences, such that similar students are subject to longer or shorter sequences depending on 
where they attend college. At CUNY, during the time period of our study, all six community 
colleges referred students with a pre-algebra score below 27 to their lowest level of 
developmental math; thus in the analysis, Hodara and Jaggars focused on students who received 
the lowest score of 17 to a score of 26. Five colleges had a two-course sequence, such that 
students scoring below 27 on the pre-algebra exam started two levels below college-level math. 
One college had a three-course sequence, such that students scoring in the same range started 
three levels below college-level math. According to course descriptions in the college catalogs, 
the two-course sequences consisted of an arithmetic and beginning algebra course, while the 
three-course sequence comprised an arithmetic course and two algebra courses. Colleges with 
the two-course sequence spent an equal or greater number of contact hours on arithmetic 
concepts (3 to 6 credits, versus 3 credits in the longer sequence), but fewer contact hours on 
algebra concepts (4 to 6 credits, versus 8 credits in the longer sequence). The course descriptions 
also suggest that the highest-level course in the three-course sequence covered slightly more 
advanced material than the algebra courses in the shorter sequences. 


22 



To disentangle the impact of sequence length from student characteristics, Hodara and 
Jaggars (2013) compared the outcomes of similar students who started in the lowest-level of the 
three-course and two-course sequence using propensity score matching and cohort fixed effects 
so that only students within the same semester of entry were compared to each other. Among the 
matched sample of students, controlling for student-level covariates and semester of entry, 
students who started in a shorter sequence were 3.5 percentage points more likely to enroll in and 
3 percentage points more likely to pass college math than their counterparts who started in the 
longer sequences. These estimates are significant at the 1 percent level. However, starting in a 
shorter sequence has no impact on three-year college credit accrual and only a small positive 
impact on credential completion. Controlling for student characteristics and cohort, those who 
started in a shorter sequence were 1 percentage point more likely to earn an associate degree over 
three years compared to their counterparts; this estimate is only significant at the 10 percent 
level. 

The Community College of Denver (CCD) began to offer compressed versions of 
developmental courses in 2005, calling the program FastStart@CCD (Edgecombe et ah, 2013a). 
FastStart encompasses a variety of course combinations but typically combines two sequential 
courses that would take two semesters to complete into one, single-semester course that 
maintains the same amount of contact hours. Edgecombe et al. (2013a) employed mixed methods 
to study the reform. Their qualitative research on FastStart found that the curriculum was also 
revised to allow for less review and more in-depth instruction on certain math topics, and that 
instructors changed their pedagogy in order to keep students engaged over longer stretches of 
course time. For example, rather than lecturing the entire time, math instructors reported 
integrating more group activities into their class. Another component of FastStart is that each 
participating student receives case management, which includes academic, career, and personal 
advising. 

Edgecombe et al.’s (2013a) study examined the effects of two pairings: Fundamentals of 
Math and Pre-algebra (MAT 030/060, worth 5 credits) and Beginning Algebra and Intermediate 
Algebra (MAT 060/090, 7 credits). The analysis compared students who took a FastStart section 
between spring 2006 and spring 2008 to students who took a stand-alone version of the first 
course in the same pairing over the same time period. Both groups were tracked through fall 
2010. Students who enrolled in the FastStart and non-FastStart sections were not equivalent: 
FastStart students were more likely to be female and Hispanic and less likely to be Black, less 
likely to have been referred to the lowest levels of developmental reading, more likely to be 
attending school full-time during the semester they enrolled in the paired course, and had slightly 
higher incomes. As a result, the authors used regression analysis to control for observable 
average differences. 

Edgecombe et al. (2013a) found that, controlling for a wide range of student-level 
characteristics, there was no difference in the semester-to-semester persistence, three-year 
persistence, credits passed, and college credits passed among FastStart and non-FastStart 
developmental math students, but FastStart had a positive impact on passing the highest 
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developmental math course (MAT090) and enrolling in and passing college math. In additional 
regressions, the authors controlled for the degree of case management the control group received, 
since less intensive case management was also available to students who did not participate in 
FastStart. The predicted proportion of FastStart students enrolling in college math was 38 
percent, and the predicted proportion passing college math was 30 percent. This compares to 26 
percent of non-FastStart students who received intensive case management enrolling and 14 
percent passing; 14 percent of non-FastStart students who received light-touch case management 
enrolling and 9 percent passing; and 22 percent who received no case management enrolling and 
16 percent passing. 

Overall, both studies (Edgecombe et al., 2013a; Hodara & Jaggars, 2013) found a 
positive impact of shorter developmental math sequences on enrolling in and passing college 
math, but little indication that acceleration has long-term impacts on students. The main 
limitation of these studies is that possible unobservable differences between students who 
selected the colleges with shorter sequences at CUNY or the compressed math courses at CCD 
and students who selected the college with the longer sequence at CUNY or the traditional math 
courses at CCD were not accounted for in the analysis. As a result, the effects may be biased 
upward if students who selected into the treatment were more advantaged across unobservable 
characteristics or biased downward if students who selected into the treatment were more 
disadvantaged across unobservable characteristics. 

A second concern of acceleration models, in general, is that compression may accelerate 
students’ progress into college math, but those who enroll may not be as prepared as their 
counterparts who completed the traditional sequence. However, both studies (Edgecombe et al., 
2013a; Hodara & Jaggars, 2013) examined conditional college math pass rates and found no 
difference in the college math pass rates of FastStart and non-FastStart students who enrolled in 
college math at CCD and students from shorter versus longer sequences who enrolled in college 
math at CUNY, indicating that the accelerated model did no hann to students’ math learning. 

Learning communities. Much of the literature on learning communities has been 
qualitative or has not focused on developmental math. The Learning Communities 
Demonstration project was a unique RCT conducted at six community colleges across the 
country; two of the colleges, Houston community college (HCC) and Queensborough 
community college (QCC), linked a developmental math course with a college-level or student 
success course (Weissman et al., 2011). 

At QCC, new students who placed into either level of developmental math were eligible 
to enroll in a learning community that linked the lowest or highest developmental math course 
with developmental English or college-level English (fall 2007) or with a college-level 
introductory course (spring 2008 and beyond) (Weissman et al., 2011). Beginning in fall 2007 
through spring 2009, 1,034 new, eligible students who were interested in joining a learning 
community at QCC were randomly assigned to a learning community or to a control group that 
had to take a regular developmental math course. A significantly higher proportion of students in 
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the treatment group completed their developmental math course in the semester they participated 
in the learning community compared to students in control group (34 percent compared to 22 
percent). But by the spring 2010 semester, students in the control group caught up, such that 
there was no difference in the developmental and college math completion rates of students in 
the treatment and control groups. The authors hypothesized that the sole impact of the QCC 
learning community may have been on encouraging students to not delay their math 
requirements. 

At HCC, new students who placed into the lowest level of developmental math were 
eligible to enroll in a learning community that linked developmental math with a student success 
course (Weissman et ah, 2011). From spring 2008 to fall 2009, 1,273 eligible students were 
randomly assigned to the treatment or control group. Findings at HCC are slightly more 
promising. A significantly higher proportion of students who participated in the learning 
communities passed the lowest level developmental math course in their program semester 
compared to students in the control group, and by spring 2010, there was still a significant 
different in treatment and control group completion rates (57.6 percent compared to 47.4 
percent). A higher proportion of learning community students also attempted the highest level 
math course compared to control students, but a similar proportion of students in the treatment 
and control groups completed the highest level developmental math course. 

At both colleges, there was no difference between the treatment and control groups in 
terms of college credit accumulation or persistence (Weissman et al., 2011). Thus, for the most 
part, the learning communities positively impacted students’ developmental math pass rates but 
had no other impact. This is not to say that learning communities, in general, have no impact on 
students; the particular features and model of the learning community are vitally important. This 
same RCT found that the learning community model at Kingsborough Community College, 
which linked developmental or college English to two courses, an orientation course and an 
academic course, and which emphasized curricular integration, advising, and student supports, 
had a positive impact on the graduation rates of the treatment group. 

Modularization. Modularization is a popular reform in developmental math, and it is 
often accompanied by computer-mediated instruction allowing students to work through the 
modules at their own pace (Twigg, 2005). Community colleges that redesign their developmental 
math courses so that the curriculum is delivered in discrete units through instructional software 
report improvements in developmental pass rates, college math enrollment and pass rates, and 
persistence for developmental math students (see, e.g., Speckler, 2008; Squires, Faulkner, & 

Hite, 2009; Twigg, 2005). However, this descriptive evidence does not utilize a comparison 
group; instead, it compares pass rates in the years before the redesign to pass rates in the years 
after the redesign is implemented, without adjusting for student differences pre- and post-course 
redesign and considering any time-varying factors, which could affect pass rates. For example, if 
enrollments increase in the years modularization is implemented, perhaps because an economic 
downturn propels more students into college, the average educational achievement of community 
college students’ may increase regardless of the educational intervention they are exposed to. Or, 
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if modularization leads to greater attrition and only the most motivated students remain, then 
average outcomes will seem higher under modularization. 

Boatman (2012) employed a more rigorous QED in her study on the impact of 
modularization at Cleveland State Community College and Jackson State Community College in 
Tennessee, colleges that redesigned developmental math in 2007 with funding from the Fund for 
the Improvement of Postsecondary Education (FIPSE) through the U.S. Department of 
Education. Both colleges divided their developmental math curriculum into 12 modules, 
although not all modules were required of students in less math-intensive degree programs. 
Students worked through the modules at their own pace in an open computer lab with the 
assistance of an instructor and tutors. 

In Tennessee, students place into developmental math, reading, and writing based on 
their ACT scores, and the state sets a cutoff score that sorts students into developmental or 
college coursework. This statewide consistency allowed Boatman (2012) to utilize a regression 
discontinuity design (RD) to identify the effects of modularizing developmental math. Because 
students do not perfectly comply with their developmental assignment, Boatman used an 
instrumental variable or "fuzzy" RD model, where the first stage predicts assignment to 
developmental education as function of ACT scores and the second stage estimates the effect of 
the predicted probability of developmental math enrollment on a set of outcomes. Boatman 
tested three different models, although I only discuss the most informative one in detail; in the 
others Boatman found null effects of moduralization. 

Boatman (2012) compared the outcomes of students in developmental math versus 
college-level math prior to the redesign and under the redesign. At Cleveland State prior to the 
redesign, students who barely placed into developmental math (i.e., were very near the cutoff) 
were 15 percentage points more likely to persist to the second semester than their counterparts in 
college math, but after the courses were modularized, developmental math students were 36 
percentage points less likely to persist to the second semester than before, or they were 2 1 
percentage points less likely to persist than college math students. This negative effect of 
modularization on persistence is perhaps due to the self-paced nature of the courses: less 
direction and structure may be related to lower persistence. At Jackson State, there were no 
differences in the outcomes of students in developmental versus college math courses pre- 
redesign and post-redesign. Overall, the grant-funded redesign of developmental math had no 
impact or a negative impact, and students would have performed the same or better under the 
traditional sequence structure. 

Less Prevalent Models of Developmental Reform 

In this section, I summarize evidence from one descriptive study on a statistics pathway 
(Hern, 2012) and two QED studies on mainstreaming (Boatman, 2012; Zeidenberg, Cho, & 
Jenkins, 2010). The two QED studies found positive impacts of mainstreaming on student 
outcomes in the trivial to small effect size range (see Appendix B). 
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Curricular Redesign. To date, the only evidence on statistics pathways for non-STEM 
students has been descriptive, with colleges comparing college math enrollment and pass rates in 
the traditional developmental math sequence versus the new statistics pathway. For example, 
Hern (2012) reported that among students who took the traditional developmental algebra 
sequence at Los Medanos College (N = 1756), a two-year college, 33 percent who placed into 
intennediate algebra eventually passed a college math course, 17 percent who placed into 
elementary algebra eventually passed, and only 9 percent who placed into arithmetic eventually 
passed college math. In stark contrast, among students who took the new developmental statistics 
course (N = 119), 82 percent of students who would have had to take intennediate algebra based 
on their placement scores passed college statistics, 78 percent who would have had to take 
elementary algebra passed, and 38 percent who would have had to take arithmetic passed. 
Although these results are promising, students who took the statistics pathway make up a very 
small sample, and this comparison does not account for differences in students who selected the 
statistics pathway versus the traditional pathway. 

Mainstreaming. Boatman’s (2012) work in Tennessee also evaluated the effects of 
mainstreaming in math. Austin-Peay State University, a four-year college, took advantage of 
FIPSE to redesign developmental math by eliminating its two developmental math courses and 
creating two college-level math courses for students whose ACT scores placed them in 
developmental math. The college offered enhanced versions of Fundamentals of Mathematics 
and Elements of Statistics, two college-level math courses, li nk ed to tutoring workshops for 
students who placed into developmental math (i.e., scored below the cutoff on the ACT that 
determines if students are college-ready in math or not). Boatman estimated the same RD models 
described previously (in the evaluation of modularization at Cleveland State and Jackson State 
community college) to evaluate mainstreaming at Austin-Peay. Unlike the effects of 
modularization, the effects of mainstreaming were generally positive. 

Boatman (2012) found no difference in the outcomes of students who barely placed into 
developmental math (very near the cutoff) who enrolled in the new college math sections and 
students who barely placed into college math (very near the cutoff). This finding is quite 
promising because it means that students who were traditionally placed in developmental math 
can do just as well as their college math counterparts when they are instead placed into college 
math sections specifically designed with their needs in mind and provided tutoring supports. 

Results were even more positive when Boatman (2012) compared the effects on students 
assigned to developmental math at Austin-Peay pre-redesign and post-redesign. Compared to 
students who barely placed into developmental math (very near the cutoff) before 
mainstreaming, students who barely placed into developmental math (very near the cutoff) who 
enrolled in the new college math courses passed more credits in their first semester, first year, 
and over two years, and they were more likely to persist from their first to second semester. 
Additionally, they were almost 7 percentage points more likely to persist to the next semester 
than students who just placed into college math, and they earned almost one more credit over 
two years than college math students. Boatman cautioned against interpreting the results as a 
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sweeping endorsement of mainstreaming since one potential interpretation of the findings is that 
cutoff scores were too high and, as a result, students scoring below the cutoff that placed them 
into developmental math were actually college-ready in math. However, the alternative 
hypothesis is that thoughtfully designed acceleration programs, such as FastStart@CCD, the 
mainstreaming model at Austin-Peay, and the I-BEST model in Washington that I will discuss 
next, may improve the math learning and college success of students compared to traditional 
developmental math sequences. 

The Washington State Board for Community and Technical Colleges (SBCTC) 
developed a mainstreaming model, called the Integrated Basic Education and Skills Training (I- 
BEST), intended to accelerate adult basic skills students’ progress through the coursework 
required of their occupational certificate or associate degree program (Zeidenberg et ah, 2010). 

In the I-BEST model, a basic skills instructor and an occupational instructor team-teach a 
college-level occupational or career-technical education (CTE) course and are in the classroom 
together for at least 50 percent of the instructional time (Wachen et al, 2012). When they are 
team-teaching, English and math basic skills content is integrated into the occupational 
curriculum, but the degree of integrated instruction varies widely across I-BEST programs. I- 
BEST programs also utilize contextualization, in which students focus on academic skills in 
support courses or in labs that are taught using applied examples from their occupational 
coursework. 

Zeidenberg et al. (2010) evaluated the effects of I-BEST using multiple methods: 
propensity score matching (PSM) and difference-in-differences (DID). The authors did not report 
on the effects of integrating math, specifically, however; math and English content were often 
integrated together in the I-BEST courses, so the extent of the focus on math could not be 
determined. Using PSM, the authors matched basic skills students who enrolled in an I-BEST 
course to basic skills students who were in an occupational program, as evidenced by enrolling in 
at least one CTE course. The students who took an I-BEST course had higher college credit 
accumulation and degree completion, but these results may be subject to selection bias since 
PSM cannot account for the unobservable differences between basic skills students who chose to 
enroll in an I-BEST course versus a traditional CTE course. 

The DID analysis provides more rigorous results because it can account for unobservable 
differences between the treatment and control groups (Zeidenberg et al., 2010). In the DID 
analysis, the authors used two groups of colleges: the treatment group included colleges that 
implemented I-BEST in the 2006-07 academic year, and the control group included colleges that 
implemented I-BEST in the 2007-08 academic year. The authors then estimated the effect of I- 
BEST enrollment from 2005-06 to 2006-07. Essentially, the DID estimate represents the 
increase in student outcomes from pre- to post-I-BEST implementation, while controlling for any 
other external changes that may have affected student outcomes during that period using similar 
colleges that implemented I-BEST in later years. Zeidenberg et al. found that basic skills 
students in occupational programs at schools where I-BEST was offered were 7.5 percentage 
points more likely to earn a certificate within three years (but no more likely to earn an associate 
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degree) and almost 10 percentage points more likely to earn some college credits compared to 
similar students at colleges without I-BEST. It is important to recognize, however, that because 
I-BEST involves the integration of both math and English basic skills, the results of the PSM and 
DID analyses cannot be attributed to math alone. 

Directions for Research on Developmental Math Reforms 

Three directions for research on developmental math reforms emerge from this review. I 
discuss the first two here; the third direction for research concerns modularization and computer- 
mediated instruction, which I discuss in the fifth section under directions for research regarding 
improving math instruction. 

First, in the area of acceleration, undertaking rigorous evaluations that address the 
mechanisms underlying acceleration may prove useful for practitioners thinking about the design 
of new accelerated math sequences. Four studies (Boatman, 2012; Edgecombe et al., 2013a; 
Hodara & Jaggars, 2013; Zeidenberg et al., 2010) have found positive effects of shorter 
developmental math sequences and mainstreaming developmental math students into college 
math classes. However, it is unclear how to ensure that acceleration models will have a lasting 
effect on the math learning and college progression of students. 

One hypothesis is that structural changes to the developmental sequence may have 
limited effects, but the combination of structural, curricular, and pedagogical changes to a 
developmental math sequence as well as the provision of non-academic supports can impact the 
college success of students in long-lasting, meaningful ways. For example, a structural change, 
such as combining two developmental math courses into one, eliminates an opportunity for 
students to exit the sequence between semesters and, as a result, automatically boosts the number 
of students completing developmental math. However, also attending to curriculum and 
pedagogy may lead to a deeper impact when the accelerated model facilitates the delivery of 
more rigorous, engaging content and standards that help motivate students struggling with math 
and may thus better prepare them for college-level work (Koski & Levin, 1998). 

A second area for future research is exploration of the implementation and impact of 
statistics pathways; Rutschow and Schnieder (2011) recommended the same direction for 
research in their review of what works in developmental education. A quasi-experimental or 
experimental evaluation of a statistics pathway at a college or set of colleges is warranted since 
there are not yet any RCT or QED studies of statistics pathways. Additionally, such a study 
might include a qualitative component that explores pedagogical innovations and non-academic 
supports within a statistics pathway and the professional development and growth tied to 
teaching in a statistics pathway compared to what is found in more traditional developmental 
math. Another important area of inquiry is student perspectives regarding how they choose the 
non-STEM statistics pathway over a more traditional developmental algebra pathway, how 
students feel the statistics pathway impacts their math learning and confidence, and the 
implications of their choices for their future academic and career goals. 
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In summary, I recommend two directions for research on developmental math reforms: 

1 . An exploration of the mechanisms underlying different models of 
acceleration. 

a. What are the comparative effects of developmental math 
acceleration models that employ: (1) only a course structure 
change, (2) both structural and curricular changes, (3) structural, 
curricular, and pedagogical changes, and (4) structural, curricular, 
pedagogical changes as well as additional academic and non- 
academic supports? 

b. Do models of acceleration have the same effect as simply lowering 
placement scores and allowing more students into college math? 

2. Quantitative and qualitative studies on statistics pathways. Research 
questions may include: 

a. Compared to enrolling in a traditional developmental algebra 
sequence, what are the causal effects of enrolling in a remedial 
statistics pathway on non-STEM students’ enrollment in and 
passing of college-level math, college progression, and degree 
completion? How do these results vary by gender, race, and other 
student demographics? 

b. How do faculty instruction and provision of non-academic 
supports compare in a developmental statistics versus algebra 
pathway? 

c. How do students choose to enroll in a statistics pathway, and what 
implications does this choice have for students who decide to 
switch majors later? 

d. What do students report are additional benefits (or drawbacks) of a 
statistics pathway compared to a developmental algebra pathway? 
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5. Strategy Four: Improving Math Instruction 

There are no national studies that document the common features of math instruction in 
developmental and college math courses. Based on qualitative studies, however, typical 
developmental math pedagogy is thought to rely largely on procedural skill-building (Goldrick- 
Rab, 2007; Hammerman & Goldberg, 2003). For example, observational studies at thirteen 
community colleges in California found that mathematics instruction was characterized by 
“remedial pedagogy,” i.e., review, lecture, independent seat-work, and math problems devoid of 
application to the real world (Grubb, 2013; Grubb & Worthen, 1999). Although traditional 
features of math instruction that emphasize procedural skills have been linked to better 
performance on standardized tests, and although much of the math that people encounter in their 
lives requires the ability to use algorithms to quickly and accurately solve computations, in order 
to truly understand mathematics, students need much more than procedural fluency (Hiebert & 
Grouws, 2007; Kilpatrick, Swafford, & Findell, 2001). 

Therefore, an important component of improving students’ college math readiness is 
improving math instruction in both the developmental and college-level classroom in order to 
support math learning and success. Kilpatrick et al. (2001) identified five interdependent strands 
of mathematical learning that instructional practices must address to build mathematical 
proficiency: 

1 . conceptual understanding — the understanding of why and when a 
mathematical idea is important or useful; 

2. procedural fluency — the ability to use procedures in the right way and for 
the right purpose; 

3. strategic competence — problem formulation and representation; 

4. adaptive reasoning — logical reasoning about mathematical relationships; 
and 

5. productive disposition — the belief that a sustained effort in learning 
mathematics will lead to greater understanding and benefit one’s life. 

This section reviews the evidence on forms of instruction that are thought to support 
components of mathematical learning beyond procedural fluency and that should thus develop 
mathematical proficiency more effectively than traditional instruction. 

Evidence on Improving Math Instruction 

Studies reviewed in this section draw primarily from a previous literature review I 
conducted on math instruction (Hodara, 2011). I also include studies from Perin’s (2011) 
literature review of contextualization and studies from a search of more recent quantitative 
literature on college math instruction. Both of the literature reviews covered pedagogical 
innovations in secondary and postsecondary schools, but for the purpose of this review, I include 
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only the studies from postsecondary math classrooms. Because of the large number of studies in 
this section (28), I only discuss the RCT and QED studies; see Appendix B for a list of all 
studies. The highest quality studies on improving math instruction at the postsecondary level 
have found positive effects of employing structured forms of student collaboration and using 
multiple representations to teach mathematics, with effect sizes ranging in the small to moderate 
range. 11 

Structured Student Collaboration 

Two of the studies on using structured forms of student collaboration in developmental 
math used a fairly rigorous design and found small to moderate effects on students’ course 
outcomes and/or math achievement (Dees, 1991; DePree, 1998), while the remaining five studies 
did not adequately adjust for pre-treatment differences between the treatment and control groups 
(Duncan & Dick, 2000; Hooker, 2010; Keynes & Olson, 2000; Norwood, 1995; Siadat, Musial, 

& Sagher, 2008). In the QED study by DePree (1998), students selected one of 50 sections of a 
developmental pre-algebra course, unaware that seven sections would utilize small-group 
instruction based on the Johnson and Johnson cooperative learning model and that six sections 
would be included in a control group and utilize the traditional lecture mode of instruction. The 
Johnson and Johnson cooperative learning model is characterized by face-to-face interaction, 
personal responsibility in working toward a shared goal, the use of interpersonal skills, and group 
processing through the exchange of feedback, explanations, and other infonnation (Johnson, 
Johnson, & Smith, 1991). In structured cooperative learning situations with these elements, 
motivational, affective, and cognitive mechanisms are thought to lead to improvements in 
learning outcomes. In the cooperative learning sections, a significantly higher percentage of 
students completed the course than in the control sections (42 percent compared to 34 percent), 
and female and Latino students in the experimental sections reported significantly greater 
confidence in their mathematical ability than those in the control sections. 

Dees (1991) randomized over 70 students in her developmental math course into four 
laboratory sections taught by graduate assistants: two that used structured, small-group 
instruction and two that used teacher-directed instruction with no group work. In the small-group 
instruction sections, particular individuals in groups of four to six students received only part of 
the instructions to a problem, and then students shared with their group the information they 
received. The group had to work together to understand the problem instructions and then solve 
the problem. At the end of the activity, one group member was randomly chosen to explain the 
group’s solution, and the group’s grade was based on this explanation, so group members had to 


1 1 It is interesting to note that many of instructional reforms have impacts that are of the same or greater size as 
programs and interventions discussed in the previous sections that require a substantial investment of resources. 
However, the outcomes in the instructional studies, which are typically related to math learning, and the outcomes in 
the other studies, which include placement test performance, enrolling and passing college math, and general college 
progression, are not necessarily comparable; the reforms may have different impacts on the long-term success of 
students. 
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collaborate to ensure everyone in the group understood the solution steps and final answer. 
Students in the cooperative learning lab consistently outperfonned students in the control group 
on measures of math achievement that included teacher-made tests and a standardized final 
exam, with effect sizes ranging from 0.39 to 0.56 standard deviations. 

Using Multiple Representations 

The next set of studies covers math instructional strategies that emphasize conceptual 
learning, a fairly abstract concept that is represented in these studies by concrete activities in 
which the instructor teaches students to solve problems in different ways using multiple 
representations of math, such as using graphing calculators, drawing graphs, creating tables in 
Excel, and using traditional fonnulas (Chappell, 2006; Hollar & Norwood, 1999; O’Callaghan, 
1998; Zawaiza & Gerber, 1993). These studies found small to large positive effects, and 
O’ Callaghan (1998) and Chappell (2006) conducted fairly rigorous studies by adjusting for the 
pre-treatment differences of the treatment and control groups. However, the validity of the 
results in the O’Callaghan (1998) study are questionable because the outcome measure, an 
assessment of mathematical learning designed by the author, is aligned directly with the 
curriculum of the treatment classrooms (also taught by the author) and, as a result, is not a fair 
assessment of the kinds of procedural skills emphasized in the control group. 

The highest quality study in this set, Chappell (2006), employed a number of methods to 
ensure that even though students self-selected into concept-based calculus and traditional 
calculus sections (unaware of the instructor or instructional method of each section), the faculty 
and students across both groups were comparable. In addition, frequent, unannounced classroom 
observations by faculty not directly involved in the study confirmed that in the concept-based 
sections, faculty taught students how to solve problems using numerical, graphical, and algebraic 
methods while constantly connecting new ideas to prior knowledge. In the control sections, 
faculty moved through the textbook teaching definitions and formulas in a linear manner. 
Students in the concept-based sections performed significantly better on the midterm and final 
exams and were better able to transfer their understanding to unfamiliar concepts. For example, 
on a final exam problem that had never been introduced in any of the classes, 88 percent of the 
students in the concept-based classrooms answered this question correctly, representing their 
answer in different ways (e.g., with an algebraic formula, graph, and/or numerically), and only 3 
percent did not support their answer with an explanation. Only 54 percent of the students in the 
traditional sections answered this question correctly, with most of them providing textbook 
formulas or definitions to explain their answer and 3 1 percent not providing any explanation. 

Emphasis on Metacognition 

The next studies explore the impact of metacognitive strategies that promote an 
awareness of each student’s own problem-solving and thought processes in developmental and 
college math (Abdalkhani & Menon, 1998; Porter, 1996; Schurter, 2002). The findings are 
mixed with moderate negative to small positive effect sizes. All three studies provide detailed 
descriptions of innovative ways in which to incorporate writing, self-reflection, and 
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comprehension monitoring into math instruction, but, in addition to not adequately controlling 
for pre-treatment differences between the treatment and control groups, in two of the studies, the 
author was the instructor of the treatment classes, but not the control classes. 

Contextualization 

The following five studies on using application-based instruction or contextualization 
generally find positive effects of this mode of instruction with trivial to small positive effect 
sizes (Ellington, 2005a; Ellington, 2005b; Ganter & Jiroutek, 2000; Shore, Shore, & Boggs, 

2004; Vasquez, 2004), but similar to the previous set, the studies have methodological 
drawbacks that call into question the validity of their findings. In Perin’s (2011) extensive review 
of contextualization (i.e., the practice of embedding discipline-based content into basic skills 
instruction) in K-12, adult literacy, and developmental classrooms, the studies, mostly from 
English classrooms, generally found positive effects, suggesting that this is a promising means of 
supporting academically underprepared students. However, only one study in Perin’s review 
(Shore et ah, 2004) was conducted in the postsecondary math classroom, so there is little 
evidence on contextualization in this setting specifically. 

Computer-Mediated Instruction 

The final category, computer-mediated instruction, includes studies that assess the impact 
of developmental math courses in which students work through technology-delivered 
mathematics content at their own pace during some or all of classroom time, with the instructor 
providing some face-to-face interaction through offering individualized attention, delivering 
instruction, and/or providing technology support (Ashby, Sadera, & McNary, 2011; Aycaster, 
2001; Garcia, 2003; Hagerty & Smith, 2005; Lovett, Meyer, & Thille, 2008; McClendon & 
McArdle, 2002; Taylor, 2008; Zavarella & Ignash, 2009; Zhu & Polianskaia, 2007). All of the 
studies in this category were conducted in classrooms in which the curriculum was not 
modularized, or divided into discrete units; studies of computer-mediated instruction in 
modularized classrooms were covered in the previous section. Additionally, I did not include 
studies that only assess the impact of fully online learning models in this category; a 
comprehensive review of online education at community colleges is covered in articles by 
Jaggars (2011, 2012), which generally find negative effects for fully online learning among 
community college students. 

The set of studies on computer-mediated instruction is notable for the high degree of 
mixed results: three studies found small positive effects of computer-mediated instruction, one 
study found small negative to moderate positive effects, and five studies found trivial to small 
negative effects. Additionally, only two of the studies employed an equivalent comparison 
group, so the evidence is inconclusive on the effects of computer-mediated instruction on student 
outcomes. 

The two studies with fairly rigorous designs both found positive effects of computer- 
mediated learning. Lovett et al. (2008) evaluated the effects of an accelerated Open Learning 
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Initiative (OLI) college statistics course at Carnegie Mellon University, in which students 
worked through interactive statistics activities delivered through computer software at their own 
pace and met with an instructor in a lab during a scheduled time. Lovett et al. (2008) randomized 
students to the eight- week OLI statistics course and a 15 -week lecture-based course and found 
positive effects of the OLI course on students’ final exam performance. However, while these 
findings suggest promising results for this model of math instruction, it is impossible to 
disentangle any effects of the OLI pedagogy (i.e., the particular computer software and the mix 
of instructor-directed and self-paced learning) from the effects of acceleration. In other words, 
the eight-week structure of the course may have contributed to differences in final exam 
outcomes either by motivating students or through another mechanism. 

Finally, the study in this section that provides the most direct evidence of the effects of 
computer-mediated instruction used a quasi-experimental design to evaluate the use of ALEKS 
in college algebra (Hagerty & Smith, 2005). Students at a university in the Midwest self-selected 
into six algebra sections without knowing which courses were ALEKS-based and which were 
traditional, textbook-based courses. Each instructor taught both a treatment and control section 
offered on the same days (except for one instructor who only taught a control section because he 
was unfamiliar with ALEKS). The students in the ALEKS sections experienced small gains in 
algebra learning compared to students in traditional algebra course sections, as measured by a 
math assessment. However, the final results only include students who took both the pretest and 
the posttest, not accounting for the test scores of students who enrolled late or withdrew from the 
course, so differential attrition could have biased the results. 

Directions for Research on Improving Math Instruction 

There are two valuable general directions for future research on instruction in 
developmental and college-level math. First, modularization and computer-mediated instruction 
are common reforms and deserve deeper understanding; Rutschow and Schnieder (2011) also 
pointed out that technology-aided instruction is extremely popular in developmental education 
yet is under-researched. 

Thus far, evidence from three sets of reviews on modularization and computer-mediated 
instruction points to a mix of results that provides no clear understanding of the effects of these 
reforms on students. First, the studies on computer-mediated instruction reviewed in this section 
found negative, positive, or null effects on student outcomes, although most studies suffer from 
methodological weaknesses, so the results are difficult to interpret. Second, many of the studies 
on modularization in the previous section found positive impacts but did not make use of an 
equivalent comparison group or any comparison group at all. In contrast, Boatman (2012) has 
presented the highest quality study on modularized developmental math and found negative 
impacts. However, it is impossible to assess if these effects were due to the modularization of the 
curriculum or the pedagogy, which includes the impact of the particular software, the self-paced 
nature of learning, and the degree of individualized attention from instructors. Third, studies 
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from the state of Washington and Virginia on online learning find negative impacts of online 
learning in math courses on student learning and college progression (Xu & Jaggars, 2011a; Xu 
& Jaggars, 2011b). These negative effects may be due to a lack of instructor guidance and 
support, difficulties with the software, and/or a lack of structure compared to face-to-face 
courses (Jaggars, 2012). These same mechanisms underlying the negative impact of online 
learning may be present in self-paced, computer-mediated classrooms in developmental math. As 
a result, an important direction for research should be rigorous evaluations of modularization and 
computer-based instruction in developmental math that identify features of these instructional 
models that support students and improve their college math readiness and success. 

A second direction for research concerns a general question about how to encourage the 
adoption and spread of effective pedagogies in developmental and college math classrooms. 
While there is still much to learn about how to teach math effectively in higher education, 
existing research, particularly from the K-12 setting, does point to some effective strategies, and 
yet there is qualitative evidence that developmental math instructors widely use “remedial 
pedagogy” (Grubb, 2013). An extension of this research area is to examine the efficacy and 
appropriateness of adopting math instructional strategies that are effective in one setting (such as 
the secondary or college-level math classroom) for another setting (such as the developmental 
math classroom). There are a number of rigorous studies from the K-12 setting that have found a 
positive impact of using structured forms of student collaboration on students’ math learning 
(Hodara, 2011). Two studies from developmental math classrooms (Dees, 1991; DePree, 1998) 
suggest that different forms of student collaboration, designed specifically for college students, 
are also effective. Therefore, the evidence seems to suggest that research-based or thoughtfully 
designed forms of student collaboration are effective in many different settings. However, in 
tenns of using multiple representations, all four studies are from the college-level math 
classroom. Thus, the question remains: If developmental math instructors adopted and adapted 
the approach used in the Chappell (2006) study for their own classroom, would they find the 
same positive effects of using multiple representations on students’ math learning? 

To summarize, I recommend two directions for research on developmental math refonns: 

1 . Identifying the effects of modularization and computer- 
mediated instruction. Research questions may include: 

a. What forms of computer-based learning support student 
math learning and college success? How do the effects of 
computer-based learning models vary depending on the 
degree of self-paced learning, teacher and student 
interaction, and student collaboration in the classroom? 

b. Do certain software programs lead to greater gains in 
student learning? If so, what are the specific instructional 
elements that seem to contribute to these gains? 
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c. To what extent are the effects of modularization on 
students’ math learning and long-tenn outcomes due to 
curricular redesign (dividing the curriculum into discrete 
units) versus pedagogical changes (the use of instructional 
software to deliver curriculum)? To what extent are the 
effects due to the degree of teacher and student interaction 
and student collaboration in the classroom? 

d. How do the effects of modularization and computer- 
mediated instruction vary for different types of students — 
for example, for students in developmental versus college 
math coursework? 

2 . Exploring ways to improve math instruction in higher 
education through the adoption and adaptation of effective 
math instructional strategies. Research questions may include: 

a. What can colleges do to encourage the adoption and 
adaptation of effective math instructional strategies in 
developmental and college math classrooms? 

b. What can developmental and college math instructors learn 
from effective math teaching at the secondary level, or 
what can developmental and college math instructors learn 
from each other? What instructional strategies that work for 
secondary math students in preparing them for college- 
level math would benefit students in developmental math or 
students struggling with college-level math? What 
instructional strategies that work for students with learning 
disabilities in math may be appropriate for students in 
developmental math or students struggling with college 
math? 
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6. Conclusion 


This review has evaluated the extent and the quality of evidence on reforms intended to 
improve the college math readiness of students entering college. Early math assessment and 
bridges, boot camps, and brush-ups target students pre-matriculation in an attempt to increase 
rates of placement into college-level math during students’ first semester in college. Reforms to 
developmental math seek to improve students’ math success in developmental math coursework 
and increase the probability of entry and success in college-level math. Improving math 
instruction in postsecondary math classrooms is important for increasing students’ math learning 
so that they can successfully complete developmental or college math coursework. Overall, some 
of these interventions and refonns appear promising, yet additional research is needed to answer 
questions that remain, including whether and how these interventions can have longer-tenn, 
more meaningful effects on students. 

Specifically, the evidence on early assessment is minimal. Two studies, a QED and a 
descriptive study, suggest that early assessment reduces the remediation rate. However, effects 
are relatively trivial, and it is unclear if early assessment has any long-tenn impact on students’ 
college success. 

The evidence on bridges, boot camps, and brush-ups suggests that short-tenn programs 
may only have short-term effects. In the RCT of the Texas summer bridge program, Barnett et al. 
(2012) found a positive effect of the bridge program on passing college math in the short-term 
but found no differences in passing college math over the long term, in persistence, or in college 
credit accumulation. The remaining five descriptive studies also found short-term effects of 
bridges, boot camps, and brush-ups in terms of increasing the number of students placing into 
college math or higher developmental courses; however, they did not address any potential 
effects on performance in college math or other longer tenn outcomes. 

The evidence on dominant models of developmental reform varies depending on the 
reform model. Research on compression is limited but positive and suggests that this reform is a 
promising means to accelerate developmental math students’ progress into college math. The 
RCT on learning communities (Weissman et al., 2011) found that the math learning communities 
did not impact students’ college math success, but effects for learning communities vary 
depending on their implementation, and it is possible that these particular programs were not 
well implemented. Finally, the QED study on modularization found that this reform had a 
negative effect on persistence and insignificant effects on all other outcomes. The negative effect 
of modularization may have been due to the self-paced, computer-mediated instructional model. 
The evidence on less prevalent developmental math refonns is positive (for mainstreaming) or 
inconclusive (for statistics pathways). 

In terms of innovations that are strictly pedagogical, the strongest evidence is in favor of 
structured forms of student collaboration and of building conceptual understanding through the 
use of multiple representations when teaching and solving problems. The evidence on computer- 
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mediated instruction in the developmental math classroom is very mixed, with some studies 
finding positive effects and others finding negative effects. Given the popularity of computer- 
mediated instruction in developmental math classrooms, an important area of research concerns 
how computer-mediated classrooms can effectively prepare students for college math and 
support their overall success. 

Based on gaps in the existing relevant findings, I outline the following directions for 
additional research: 

1 . Carrying out additional rigorous research on the effects of early 
assessment and the mechanisms underlying its effects; 

2. Building a better understanding of the role early assessment plays 
in strengthening secondary and postsecondary partnerships; 

3. Identifying cost-effective features and models of bridges and boot 
camps that lead to long-term benefits for students; 

4. Conducting causal research on the effects of test preparation 
delivered primarily through boot camps and brush-ups; 

5. Investigating the mechanisms underlying different models of 
acceleration; 

6. Conducting quantitative and qualitative studies on statistics 
pathways; 

7. Identifying the effects of modularization and computer-mediated 
instruction; and 

8. Exploring ways to improve math instruction in higher education 
through the adoption and adaptation of effective math instructional 
strategies. 
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Appendix A: Search Strategy and Inclusion Criteria 


Strategy Inclusion Criteria 


Strategy 1 


Strategy 2 


Strategy 3 


Strategy 4 


1. CCRC reports: 

a. Hodara et al. (2012) 

b. Kerrigan & Slater (2010) 

2. Education Full Text and Google Scholar search: 

a. Publication range from 2000 to the present 

b. Search terms: 

college AND early assessment program 

high school AND college AND ( developmental OR remedial ) AND ( testing OR 
exams OR assessment ) 

3. Exclude article if there is no quantitative outcome related to math or college performance. 

1. NCPR Report: Barnett et al. (2012) 

2. Education Full Text and Google Scholar search: 

a. Publication range from 2000 to the present 

b. Search terms: 
summer bridge 

college AND ( bridge OR boot camp OR brush-up OR brush up ) 

3. Exclude article if there is no quantitative outcome related to math or college performance. 

1. NCPR/CCRC reports: 

a. Report on Scaling Innovation reforms (Edgecombe et al., 2013b) 

b. Evaluation of l-BEST (Zeidenberg et al. 2012) 

c. Evaluation of FastStart (Edgecombe et al., 2013) 

d. Evaluation of Tennessee course redesign (Boatman, 2012) 

e. Evaluation of learning communities (Weissman et al., 2011) 

2. Recent literature not covered by above reviews using Education Full Text and Google 
Scholar searches: 

a. Publication range from 2011 to the present. 

b. Search terms: developmental math 

3. Exclude article if there is no quantitative outcome related to math or college performance. 

1. CCRC reports: 

a. Math pedagogy literature review (Flodara, 2011) 

b. Contextualization literature review (Perin, 2011) 

2. Recent literature not covered by above reviews using Education Full Text and Google 
Scholar searches: 

a. Publication range from 2011 to the present. 

b. Search terms: college math instruction 

3. Exclude article if there is no quantitative outcome related to math or college performance. 
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Appendix B: Summary of Studies 


Study 

Summary of Findings 

Effect 

Size 1 

Effect 

Category 2 

Non- 

Equivalent 

Comparison 

Group 

No 

Comparison 

Group 

Strategy One: Intervening During High School With Early Assessment 

Howell et al. (2010) 

At California State University-Sacramento, participation in the Early Assessment 
Program (i.e., taking the optional CSU test items on the California standardized 
test as a high school junior) reduced the probability of placing into math 
remediation as a college freshman. 

-0.037 

Trivial 

negative 



Kerrigan & Slater (2010) 

At El Paso Community College, a comprehensive college readiness protocol for 
high school students increased the proportion of students placing into college 
math and higher levels of developmental math. 

N/A 

Trivial 

positive 

X 


Strategy Two: Intervening Pre-Matriculation With Bridges, Boot Camps, and Brush-Ups 

Bridges 

Barnett et al. (2012) 

The Texas Developmental Summer Bridge Program had a positive impact on 
first college-level course completion in math and writing, but no significant 
effect on credit attainment or persistence. 

-0.076 to 
0.05 

Trivial 
negative to 
trivial 
positive 



Kallison & Stader (2012) 

The math placement scores of Texas Developmental Summer Bridge Program 
participants decreased at some colleges and increased at others, but 
gains/losses are not significant. (Effect size of 0.92 represents large gains in 
ACCUPLACER scores for six students at one college.) 

-0.007 to 
0.92 

Trivial 
negative to 
moderate 
positive 


X 

Gleason et al. (2010) 

Participants in the Engineering Math Advancement Program at University of 
Alabama increased their math placement scores and had higher retention than 
non-participant STEM students. 

N/A 

Trivial to 
moderate 
positive 

X 


Reisel et al. (2012) 

Participants in the summer bridge program at the University of Wisconsin- 
Milwaukee experienced gains in their placement scores, especially in the face- 
to-face model (compared to the online model). 

0.05 to 
1.04 

Trivial to 
moderate 
positive 


X 
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Study 

Summary of Findings 

Effect 

Size 1 

Effect 

Category 2 

Non- 

Equivalent 

Comparison 

Group 

No 

Comparison 

Group 

Boot camps 

Sherer & Grunow (2010) 

The PREP program in El Paso Community College increased the proportion of 
students placing into higher levels of developmental math. 

N/A 

Small 

positive 


X 

Brush-ups 

Hodara et al. (2012) 

At a North Carolina community college, about 35 percent of students who took 
the placement test review tested at least one level higher in developmental 
math. 

N/A 

Moderate 

positive 


X 

Strategy Three: Reforming Developmental Math 

Compression 

Hodara & Jaggars (2013) 

At the City University of New York community colleges, enrolling in a shorter 
developmental math course sequence compared to a longer math sequence 
had a positive effect on enrolling in and passing college math, and had a small 
impact on degree attainment. 

0.01 to 
0.10 

Trivial 

positive 



Edgecombe et al. (2013) 

At the Community College of Denver, FastStart students in compressed math 
courses had higher gatekeeper math enrollment and pass rates than similar 
non-FastStart students. 

N/A 

Small 

positive 



Learning communities 

Weismann et al. (2011) 

Learning communities at Queensborough and Houston community colleges led 
to more students passing developmental math courses in the program 
semester but had no other significant impacts. 

Oto 0.17 

Trivial 

positive 



Modularization 

Boatman (2012) 

At Cleveland State and Jackson State community colleges, in comparison to the 
outcomes of students around the margin of the cutoff pre-redesign, after the 
implementation of modularization, students who placed into developmental 
math had lower rates of persistence. Otherwise there was no difference in 
outcomes. 

N/A 

Small 

negative to 
trivial 
positive 



Speckler (2008) 

At 18 colleges, retention and pass rates, course enrollments, and/or grades 
were generally higher for students in sections that used MyMathLab or MathXL 
than in sections that used traditional (lecture) instruction. 

N/A 

Small to 
moderate 
positive 

X 


Squires et al. (2009) 

At Cleveland State Community College, after the introduction of 
modularization, course completion rates in developmental math and 
subsequent college-level math courses increased. 

N/A 

Small 

positive 

X 
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Study 

Summary of Findings 

Effect 

Size 1 

Effect 

Category 2 

Non- 

Equivalent 

Comparison 

Group 

No 

Comparison 

Group 

Twigg (2005) 

Different outcomes were examined at 30 selected institutions, with in-depth 
case studies on 15 institutions. Colleges that modularized their math courses 
reported increases in retention, math learning, and course pass rates, and 
reported a decreased cost per student. 

N/A 

Small 

positive 


X 

Curricular redesign (statistics pathways) 

Hern (2012) 

At Los Medanos College, students who took the accelerated statistics pathway 
had higher college math completion rates than students who took the 
traditional developmental algebra pathway. 

N/A 

Moderate 

positive 

X 


Mainstreaming 

Boatman (2012) 

At Austin-Peay State University, compared to the outcomes of students prior to 
the implementation of mainstreaming, students assigned to developmental 
math who enrolled in the new college math courses passed more credits and 
were more likely to persist from their first to second semester. 

N/A 

Trivial to 
small 
positive 



Zeidenberg et al. (2010) 

Students in occupational programs at schools where l-BEST was offered were 
more likely to earn a certificate within three years (but no more likely to earn 
an associate degree) and more likely to earn some college credits compared to 
similar students at colleges without l-BEST. 

N/A 

Trivial to 
small 
positive 



Strategy Four: Improving Math Instruction 

Student collaboration 

Dees (1991) 

Students randomized into cooperative learning sections performed significantly 
better on the algebra word problem and geometry proof-writing than students 
in the traditional sections. 

0.39 to 
0.56 

Small 

positive 



DePree (1998) 

A significantly higher percentage of students in the cooperative learning 
sections completed the course than students in the control sections, and Latino 
and female students in cooperative learning classes had positive gains in self- 
reported math confidence relative to the control group, but there were no 
differences in the achievement gains of treatment and control groups. 

0.45 to 
0.72 

Small to 
moderate 
positive 



Duncan & Dick (2000) 

The Math Excel students attained significantly higher grades than the non-Math 
Excel students, and students in the Math Excel Program outperformed their 
predicted grades, as determined by their SAT Math score, by half a grade point. 

N/A 

Moderate 

positive 

X 


Hooker (2010) 

Students in pre-algebra classes that used peer-led team leader (PLTL) 
workshops had higher persistence and completion rates than control group 
students. 

N/A 

Trivial to 
small 
positive 

X 
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Study 

Summary of Findings 

Effect 

Size 1 

Effect 

Category 2 

Non- 

Equivalent 

Comparison 

Group 

No 

Comparison 

Group 

Keynes & Olson (2000) 

Students in the Calculus Initiative classrooms had higher GPAs, pass rates, and 
retention rates than students in traditional calculus. 

N/A 

Small 

positive 

X 


Norwood (1995) 

Compared to students who took developmental algebra in the semester when 
instructors used traditional methods, a higher proportion of students who took 
developmental algebra in the semester that the instructors used the learning 
model of cooperative learning completed their first college-level math course. 

N/A 

Small 

positive 

X 


Siadat et al. (2008) 

The Keystone Method involves daily assessment of students and use of 
cooperative learning groups. Students in the Keystone Method classes had 
higher final exam scores and persistence rates than students in traditional, 
control classes. 

N/A 

Small to 
large 
positive 

X 


Using multiple representations 

Chappell (2006) 

Students in the concept-based calculus sections scored significantly better on 
the midterm and final exams than students in the traditional sections, except 
for on the final procedural skill section. 

0.34 to 
0.64 

Small to 
moderate 
positive 



Hollar & Norwood (1999) 

Scores on the function test were higher for the graphing-approach group than 
for the control group, but there were no significant differences in terms of final 
exam scores (scores not reported). 

1.02 

Large 

positive 

X 


O'Callaghan (1998) 

The Computer-Intensive algebra students scored higher than students in the 
traditional algebra classrooms on the function test, and there were no 
significant differences in final exam performance (scores not reported). 

0.86 to 
1.07 

Large 

positive 



Zawaiza & Gerber (1993) 

Community college students with learning disabilities who received a schema- 
based intervention made greater gains on a word problem test than those who 
did not, and they performed at almost the same level as their math-competent 
peers. 

0.27 to 
1.11 

Small to 
large 
positive 

X 


Metacognitive strategies 

Abdalkhani 
& Menon 
(1998) 

In earlier courses in which journal writing was not used in math, students had a 
mean score of 65 percent on quizzes, while students in the course that 
incorporated journal writing had a mean quiz score of 72 percent. 

N/A 

Trivial 

positive 

X 


Porter (1996) 

There were no differences in the number of procedural errors made by 
students in a college calculus course that used writing to learn math and 
students in a comparison course on a final exam, but students in the treatment 
group made more conceptual errors. 

-0.63 to 
0.09 

Moderate 
negative to 
trivial 
positive 

X 
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Study 

Summary of Findings 

Effect 

Size 1 

Effect 

Category 2 

Non- 

Equivalent 

Comparison 

Group 

No 

Comparison 

Group 

Schurter (2002) 

Students who received direct instruction in the use of comprehension 
monitoring or Polya's four-step problem-solving method performed better in 
mathematical problem solving than those who did not. 

N/A 

Small 

positive 

X 


Application/Contextualization 

Ellington (2005a) 

Students in modeling sections had higher levels of self-reported confidence, 
lower levels of anxiety, and lower withdrawal rates than students in traditional 
sections. 

N/A 

Trivial to 
small 
positive 

X 


Ellington (2005b) 

Students in modeling sections performed significantly better on an assessment 
and had higher pass rates than students in traditional sections. 

0.41 

Small 

positive 

X 


Ganter & Jiroutek (2000) 

Calculus sections that utilized long-term projects in the computer lab did not 
perform better than the control sections on the final exam. On the 
standardized exam the control group outperformed the treatment group. 

N/A 

Trivial 

negative 

X 


Shore et al. (2004) 

Allied health students who participated in a developmental math intervention 
using problem-based learning in which instruction was contextualized in a 
nursing topic (kidney function) showed greater gain than students in traditional 
instruction. 

N/A 

Trivial to 
small 
positive 

X 


Vasquez (2004) 

A greater percentage of students passed higher level math courses after the 
introduction of the Algorithmic Instructional Technique (AIT). AIT involves real- 
world problem-solving and content that integrates content relevant to 
students' lives. 

N/A 

Trivial to 
small 
positive 


X 

Computer-mediated instruction 

Ashby et al. (2011) 

Students had higher developmental math pass rates in the face-to-face course 
compared to online course or blended course; students in the online course 
had higher developmental math pass rates than those in the blended course. 

-0.30 to 
0.54 

Small 

negative to 
moderate 
positive 

X 


Aycaster (2001) 

The success rates in 15 developmental math courses were not related to the 
method of instruction (lecture or computer-aided instruction). 

N/A 

Trivial 

negative 

X 


Garcia (2003) 

Pre- to posttest mean ACCU PLACER scores increased for students in an 
elementary algebra class with learning style and attitude surveys, workshops, 
computer-based instruction, self-assessment, and supplemental instruction. 

N/A 

Small 

positive 


X 

Hagerty & Smith (2005) 

The students using ALEKS had higher gains from pre- to posttest than the 
students in the traditional classrooms. 

0.49 

Small 

positive 
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Study 

Summary of Findings 

Effect 

Size 1 

Effect 

Category 2 

Non- 

Equivalent 

Comparison 

Group 

No 

Comparison 

Group 

Lovett et al. (2008) 

Students randomly assigned to an eight-week accelerated OLI-Statistics hybrid 
course (in which students met with the instructor to review and reinforce 
material) performed better on the final exam than students randomly assigned 
to the traditional 15-week statistics course. 

N/A 

Small 

positive 



McClendon & McArdle 
(2002) 

Retention was higher in courses that used the lecture mode of instruction 
versus the computer software, ALEKS and Academic Systems. 

N/A 

Trivial to 
moderate 
negative 

X 


Taylor (2008) 

The control group made larger gains from pre- to posttest than the ALEKS 
group on the algebra test, but self-reported math anxiety decreased more for 
the ALEKS group than for the control group, and self-reported attitudes about 
math improved for the ALEKS group and worsened for the control group. 

-0.21 to 
-0.12 

Small 

negative 

X 


Zavarella & Ignash 
(2009) 

Twenty percent of students in the lecture-based course and 42 percent of 
students in the blended-online course withdrew from the course by the end of 
the semester. 

N/A 

Small 

negative 

X 


Zhu & Polianskaia (2007) 

Over most years in a ten-year period, a higher percentage of students in lecture 
courses had higher pass rates, course completion rates, and final exam scores 
than students in computer-mediated courses. 

N/A 

Trivial to 
small 
negative 

X 



Table Notes 

1. Effect Size: Effect sizes are used to compare results across studies with different outcome measures and are also important in measuring meaningful changes in outcomes that 
may not necessarily be statistically significant. The table reports Cohen's d effect sizes, which can be interpreted as the standardized difference between the treatment and 
control group means. The author calculated Cohen's d based on the formulas from Thalheimer and Cook (2002). Typically, Cohen's d was calculated using means and standard 
deviations for the treatment and control groups as provided in the article, using the following formula, where x is the mean, n is the sample size, s is the standard deviation, and 
the subscripts t and c denote treatment and control. If the article did not have a comparison group but provided pre- and posttest means and standard deviations, the same 
formula was used: 


d- 


x—x 

t c 


( n,-l)s t 2 +(n c -l)s c 2 


If the means and/or standard deviations were not provided but the F-statistic was, Cohen's d was calculated using F and the sample sizes of the treatment and control groups: 


d=\F\ 


»r+», 

77, 77 


V F + n c ' 


V "t"c d\"t 
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If the means and/or standard deviations were not provided but the t-statistic was, Cohen's d was calculated using t and the sample sizes of the treatment and control groups: 
d= t 

For articles that do not report the necessary information to calculate Cohen's d, "N/A" is written in the effect size column. 

2. Effect Category: Hopkins (2009) proposes a scale that can be used to compare traditional effect size estimates to differences in percentages, which most articles do report, 
allowing for an estimation of their effect size category. The following scale was used to compare effect size estimates and percentage differences, in order to categorize them by 
size: 


Size: 

Trivial 

Small 

Moderate 

Large 

Very Large 

Nearly Perfect 

Standardized difference in means [d) 

0-0.2 

0.2-0. 6 

0.6-1. 2 

1. 2-2.0 

2. 0-4.0 

4.0-°° 

Percentage difference 

0-10 

10-30 

30-50 

50-70 

70-90 

90-100 
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